Big Data Engineer
Tasks
- Automate build test and deploy with CI CD
- Build batch CDC streaming data pipelines
- Create dimensional models for Power BI datasets and APIs
- Design secure governed data pipelines
- Embed data quality rules and validation checks
- Enforce RBAC secrets management PII classification and retention
- Handle SCD partitioning and performance tuning
- Implement data contracts and schema versioning
- Instrument data lineage and end to end monitoring
- Integrate OT data via OPC UA and MQTT
- Model curated and semantic layers
- Optimize cost and performance and support FinOps reviews
- Provide documentation runbooks and post incident reviews
Perks/Benefits
- 401k matching
- Dental insurance
- Disability insurance
- Employee assistance program
- Health insurance
- Health savings account
- Life insurance
- Paid Holidays
- Paid vacation
- Training and development
- Tuition reimbursement
- Vision insurance
- Wellness programs
Skills/Tech-stack
ADLS | Alerting | Automated testing | Autoscaling | Azure Data | Azure Data Factory | Azure Databricks | Azure Key Vault | Azure SQL | Azure Synapse | Batch Frames | Batch Processing | Big Data | Blue-Green Deployment | Blue/green | CI/CD | Caching | Change Data Capture | Clustering | Clustering Sizing | Completeness Checks | Cost Optimization | Data Capture | Data Contracts | Data Engineering | Data Factory | Data Modeling | Data Pipelines | Data Quality | Data Validation | Dimensional Modeling | Dimensional models | Event Frames | Fabric | Feature Stores | File Formats | Freshness Checks | Git | Historian | ISA-95 | ISA-99 | Key Vault | Lineage | Lineage Catalogs | MDM | Metric Stores | Monitoring | Mqtt | OPC UA | Observability | On Call Runbooks | On-Call | OneLake | PII | Partitioning | Performance Tuning | Power BI | PySpark | Python | RBAC | Retention policies | Row Level Security | SAP Datasphere | SAP S4HANA | SQL | SQL MI | Schema Expectations | Schema versioning | Secrets management | Security | Semantic Modeling | Series data | Slowly Changing Dimensions | Spark Structured Streaming | Streaming | Structured Streaming | Time Series | Time Series Data
Education
Bachelor of Engineering | Bachelor of Science | Master of Science | PhD
Roles
Related jobs
-
Featured Feat. AI Engineer (MTS) USD 160K-300KAPI Development | AWS | Amazon Web Services | Deep learning | FastAPIMentoring | Open source contributions | Remote workMid-levelRemote R12d ago
-
Featured Feat. Data Engineer USD 80K-150KData Monitoring | Data Quality | Data Validation | ELT | ETLRemote workEntry-levelRemote R12d ago
-
Mid-level Full Time北京 R8h ago
-
Mid-Level Data Engineer USD 90K-98KAPI Development | Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake StorageRemote workMid-level Full TimeWork from home, VA, United States R10h ago
-
Senior Data Engineer USD 165K-180KAPIs | Anomaly Detection | Azure | Azure Data | Azure Data FactorySenior-level Full TimeWork from home, VA, United States R10h ago
-
AWS Glue | AWS Lambda | Airflow | Amazon S3 | AzureRemote workSenior-level Full TimeRemote R12h ago
-
Data Engineer (MS) (Remote) INR 2040K-3380KCI/CD | Data Transformation | Data Validation | Date normalization | ETLMentorship opportunities | Professional growth | Remote workSenior-level Full TimeMaharashtra, Pune, India R12h ago
-
AI / LLM Integration Engineer (Remote) INR 1800K-2800KAPI Integration | Anthropic SDK | Async APIs | Benchmarking | Confidence scoring100 percent remote | Collaborative culture | Inclusive team culture | Professional growthMid-level Full TimeMaharashtra, Pune, India R12h ago
-
ML / LLM Engineer (Remote) INR 2500K-3000KAmazon Web Services | Azure | Classification | Feature Engineering | Language ModelsRemote workMid-level Full TimeMaharashtra, Pune, India R12h ago
-
Evergreen - Mathematics for Machine Learning USD 80K-300KAutodiff | JAX | Linear Algebra | Machine Learning | Matrix OperationsPart-time workMid-level Full TimeRemote, Remote, BR R13h ago
-
C++ | Cloud platform | Data Pipelines | ETL | Google CloudCDI | Career growth opportunities | Flexible work environment | Telework 1 day per weekSenior-level Full TimeCastelnaudary, France R14h ago
-
Maitrise Ouvrage/Support Booking & Risk H/F EUR 21K-21KOffice 365 | Python | SQLAgile team environment | Career coaching | HR support | Modern campus services | Remote work opportunityEntry-level Full TimeEurope, France, Ile-de-France, 92 - Hauts-De-Seine R16h ago
-
Data Engineer Databricks (H/F) EUR 47K-55KAmazon Web Services | Apache Spark | Azure | Azure DevOps | CI/CDCareer development | Flexible remote work | Meal tickets | Paid time off | RTT daysSenior-level Full TimeSAINT OUEN, France R18h ago
-
Analytics Engineer EUR 48K-78KAmazon Redshift | Cloud platform | DBT | Data Modeling | Data TestingAnnual blood tests | Company apartments | Extra paid vacation days | Flexible schedule | Insurance and wellness programsMid-level Full TimeKaunas, Lithuania R18h ago
-
Senior Databricks EUR 46K-55KAWS | Apache Spark | Azure | Azure DevOps | Batch ProcessingCareer coaching | Conference speaking opportunities | Flexible telework | Meal tickets | Paid time offSenior-level Full TimeSAINT OUEN, France R18h ago
-
Anthropic Claude | Async Programming | ChatGPT | Claude Code | CodexDirect user impact | Flexible work schedule | Occasional international travel | Work from home optionMid-level Full TimeSlovenia / Remote R19h ago
-
Data Operations Engineer INR 2040K-3380KApache Airflow | Data Pipelines | Data Refresh | Data Warehousing | Data pipelineEmergency incident response support | On-call rotationSenior-level Full TimeBengaluru, INDIA, India R20h ago
-
APIs | Anomaly Detection | Data Modeling | Data Pipelines | Docker100% remote work | Career growth opportunities | Flexible work environmentMid-level Full TimeEstonia R21h ago
-
APIs | Anomaly Detection | Data Modeling | Data Pipelines | Docker100 percent remote work | Autonomous work environment | Career growth | Flexible work environment | International team cultureMid-level Full TimeHungary R21h ago
-
Anomaly Detection | Data Modeling | Data Pipelines | Docker | ForecastingCareer growth | Flexible work environment | Remote workMid-level Full TimeFinland R21h ago
-
Anomaly Detection | Data Modeling | Data Pipelines | Docker | JavaScript100% remote work | Career growth opportunities | Flexible work environmentMid-level Full TimeCzechia R21h ago
-
APIs | Anomaly Detection | Data Modeling | Data Pipelines | Docker100% remote work | Career growth opportunities | Flexible work environment | International team cultureMid-level Full TimeNorway R21h ago
-
API Integration | Anomaly Detection | Data Modeling | Docker | Machine Learning100 percent remote work | Autonomy | Career growth | Flexible work environment | International team cultureMid-level Full TimeLuxembourg R21h ago
-
API Integration | Anomaly Detection | Data Modeling | Data Pipelines | DockerCareer growth | Flexible schedule | International team culture | Remote workMid-level Full TimeCroatia R21h ago
-
APIs | Anomaly Detection | Data Modeling | Data Pipelines | Docker100% remote work | Autonomy | Career growth | Flexible work environment | International team cultureMid-level Full TimeBulgaria R21h ago