Big Data Engineer
Tasks
- Automate build test and deploy with CI CD
- Build batch CDC streaming data pipelines
- Create dimensional models for Power BI datasets and APIs
- Design secure governed data pipelines
- Embed data quality rules and validation checks
- Enforce RBAC secrets management PII classification and retention
- Handle SCD partitioning and performance tuning
- Implement data contracts and schema versioning
- Instrument data lineage and end to end monitoring
- Integrate OT data via OPC UA and MQTT
- Model curated and semantic layers
- Optimize cost and performance and support FinOps reviews
- Provide documentation runbooks and post incident reviews
Perks/Benefits
- 401k matching
- Dental insurance
- Disability insurance
- Employee assistance program
- Health insurance
- Health savings account
- Life insurance
- Paid Holidays
- Paid vacation
- Training and development
- Tuition reimbursement
- Vision insurance
- Wellness programs
Skills/Tech-stack
ADLS | Alerting | Automated testing | Autoscaling | Azure Data | Azure Data Factory | Azure Databricks | Azure Key Vault | Azure SQL | Azure Synapse | Batch Frames | Batch Processing | Big Data | Blue-Green Deployment | Blue/green | CI/CD | Caching | Change Data Capture | Clustering | Clustering Sizing | Completeness Checks | Cost Optimization | Data Capture | Data Contracts | Data Engineering | Data Factory | Data Modeling | Data Pipelines | Data Quality | Data Validation | Dimensional Modeling | Dimensional models | Event Frames | Fabric | Feature Stores | File Formats | Freshness Checks | Git | Historian | ISA-95 | ISA-99 | Key Vault | Lineage | Lineage Catalogs | MDM | Metric Stores | Monitoring | Mqtt | OPC UA | Observability | On Call Runbooks | On-Call | OneLake | PII | Partitioning | Performance Tuning | Power BI | PySpark | Python | RBAC | Retention policies | Row Level Security | SAP Datasphere | SAP S4HANA | SQL | SQL MI | Schema Expectations | Schema versioning | Secrets management | Security | Semantic Modeling | Series data | Slowly Changing Dimensions | Spark Structured Streaming | Streaming | Structured Streaming | Time Series | Time Series Data
Education
Bachelor of Engineering | Bachelor of Science | Master of Science | PhD
Roles
Related jobs
-
Early-Career Network Engineer (RAN Optimization) USD 85K-130K4G | 5G | Automation | C Band | CBRS401k match | Dental insurance | Disability insurance | Educational assistance | Financial wellness programsMid-level Full TimePlano,Texas,United States R4h ago
-
Data Engineer USD 126K-208KAPI Integration | Airflow | Amazon Web Services | BigQuery | CCPADEI initiatives | Dental benefits | Employee rewards program | Medical benefits | Mental health supportMid-level Full TimeRemote, United States R4h ago
-
AWS | Data Pipelines | Databricks | Medallion Architecture | PySparkFlexible working arrangements | Hybrid work model | International team | Remote work options | Training programSenior-level ContractVilnius, Lithuania (Remote) R5h ago
-
Senior-level Full TimeRemote R5h ago
-
Alerting | Ansible | Bash | CI/CD | CephRemote workSenior-level Full TimeUnited States, United States R6h ago
-
Ansible | Bash | CI/CD | CentOS | CephContract-to-hire | No sponsorship | Remote workSenior-level Full TimeUnited States, United States R6h ago
-
Machine Learning Engineer USD 131K-178KAWS | Cassandra | Convolutional Neural Networks | Data Lakes | Data PipelinesMid-level Full TimeRemote, NY, US R7h ago
-
GCP Data Engineer / Consultant Specialist INR 1500K-2000KAirflow | Alerting | Apache Beam | Automation | BigQueryFlexible working | Inclusive workplace | Opportunities for growth | Professional developmentMid-level Full TimePune, Maharashtra, India R8h ago
-
Administrador de Dados - Engenharia e modelagem BRL 21K-23KCA Gen | DB2 | DCL | DDL | DMLDental plan | Gympass | Health insurance | Life insurance | Meal or food voucherEntry-level Full TimeRemoto, Brasil R8h ago
-
Case design | Data Integrity | Database Testing | Database optimization | Defect ReportingSenior-level Full TimeAustin, TX R8h ago
-
BE Software Engineer USD 100K-145KCI/CD | Distributed Systems | Docker | GraphQL | JavaRemote workMid-level Full TimeRemote R8h ago
-
Senior Data Engineer USD 132K-167KAWS | DBT | Data Modeling | Data Pipeline Monitoring | Data QualityRemote workSenior-level Full TimeRemote R8h ago
-
Sr Data Engineer USD 125K-165K.NET | API Gateway | API Management | Access Control | BigQueryRemote positionSenior-level Full TimeRemote R9h ago
-
Consultant Data Scientist IA (H/F) EUR 46K-55KAzure | Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake StorageEarly access to new Microsoft technologies | End-of-year bonus | Flexible telework | Health insurance | Meal ticketsSenior-level Full TimeAix-en-Provence, Provence-Alpes-Côte d'Azur, France R9h ago
-
Software Engineer, Machine Learning USD 213K-293KAPI Design | Agent Orchestration | Artificial Intelligence | Bias Mitigation | C++Senior-level Full TimeSunnyvale, CA | Remote, US | … R10h ago
-
Data Engineer GBP 30K-30KAI tools | AWS | AWS CDK | AWS Cloud | AWS cloud infrastructureBike loan scheme | Discounted private healthcare | Employee assistance programme | Enhanced family leave | Free onsite gymEntry-level Full TimeManchester / Hybrid, England, United Kingdom R12h ago
-
Continuous Delivery | Deep learning | Integration Testing | Java | JavaScriptAnnual offsite | Coworking stipend | Learning time | Monthly in person rituals | Remote workSenior-level Full TimeFrance, France R13h ago
-
Data Engineer DBT & Snowflake - Confirmé (H/F) EUR 50K-65KAirflow | DBT | GitHub | Python | SnowflakeEmployee stock ownership | Health insurance 80 percent covered | Hybrid work | Maternity return 4 5 with pay for 6 months | Paid training coursesMid-level Full TimeBordeaux, Nouvelle-Aquitaine, France R13h ago
-
Consultant Data Scientist IA (H/F) EUR 46K-55KAzure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake Storage | Azure FabricFlexible remote work | Great place to work community | Health insurance | Meal vouchers | Mobility allowanceSenior-level Full TimeValbonne, Provence-Alpes-Côte d'Azur, France R13h ago
-
Airflow | Atlas | Confluence | Datalake | DremioMeal vouchers | Remote work | Subsidized social and cultural activities | Training opportunitiesMid-level Full TimeMérignac, Nouvelle-Aquitaine, France R13h ago
-
AWS | DBT | Data Modeling | Data Pipelines | Data QualityCollaborative environment | End to end data engineering ownership | Engineering quality focus | Fully remote | Latin America location requirementSenior-level Full TimeBrazil R16h ago
-
Senior AI Data Engineer USD 155K-185KApache Airflow | Apache Spark | Azure Synapse | BigQuery | ClickHouseEmployer paid Medical Dental Vision Insurance | Flexible paid time off | Manager check ins | Paid cell phone and service | Paid parental leaveSenior-level Full TimeRemote - United States R19h ago
-
Senior Staff Software Engineer - Data Platform USD 200K-250KAWS Glue | AWS IAM | Amazon EMR | Amazon S3 | AmundsenDevelopment dollars | Employee stock purchase program | Family-forming benefits | Financial coaching | Flexible time offSenior-level Full TimeRemote, USA R19h ago
-
Senior Staff Software Engineer - Data Platform USD 200K-250KAWS EMR | AWS Glue | AWS IAM | AWS S3 | Apache AirflowDevelopment dollars | Financial coaching | Flexible remote work | Flexible time off | Free therapy sessionsSenior-level Full TimeRemote, USA R20h ago
-
GenAI Engineer - Staff - EY GDS Spain - Hybrid EUR 58K-79KAPI Development | AWS | Agentic AI | Autogen | AzureContinuous learning programs | Hybrid work model | Psychological support | Recognition programs | Training and development programsSenior-level Full TimeMalaga, ES, 29590 R20h ago