Big Data Engineer
Tasks
- Automate build test and deploy with CI CD
- Build batch CDC streaming data pipelines
- Create dimensional models for Power BI datasets and APIs
- Design secure governed data pipelines
- Embed data quality rules and validation checks
- Enforce RBAC secrets management PII classification and retention
- Handle SCD partitioning and performance tuning
- Implement data contracts and schema versioning
- Instrument data lineage and end to end monitoring
- Integrate OT data via OPC UA and MQTT
- Model curated and semantic layers
- Optimize cost and performance and support FinOps reviews
- Provide documentation runbooks and post incident reviews
Perks/Benefits
- 401k matching
- Dental insurance
- Disability insurance
- Employee assistance program
- Health insurance
- Health savings account
- Life insurance
- Paid Holidays
- Paid vacation
- Training and development
- Tuition reimbursement
- Vision insurance
- Wellness programs
Skills/Tech-stack
ADLS | Alerting | Automated testing | Autoscaling | Azure Data | Azure Data Factory | Azure Databricks | Azure Key Vault | Azure SQL | Azure Synapse | Batch Frames | Batch Processing | Big Data | Blue-Green Deployment | Blue/green | CI/CD | Caching | Change Data Capture | Clustering | Clustering Sizing | Completeness Checks | Cost Optimization | Data Capture | Data Contracts | Data Engineering | Data Factory | Data Modeling | Data Pipelines | Data Quality | Data Validation | Dimensional Modeling | Dimensional models | Event Frames | Fabric | Feature Stores | File Formats | Freshness Checks | Git | Historian | ISA-95 | ISA-99 | Key Vault | Lineage | Lineage Catalogs | MDM | Metric Stores | Monitoring | Mqtt | OPC UA | Observability | On Call Runbooks | On-Call | OneLake | PII | Partitioning | Performance Tuning | Power BI | PySpark | Python | RBAC | Retention policies | Row Level Security | SAP Datasphere | SAP S4HANA | SQL | SQL MI | Schema Expectations | Schema versioning | Secrets management | Security | Semantic Modeling | Series data | Slowly Changing Dimensions | Spark Structured Streaming | Streaming | Structured Streaming | Time Series | Time Series Data
Education
Bachelor of Engineering | Bachelor of Science | Master of Science | PhD
Roles
Related jobs
-
Featured Feat. Software Engineer USD 40K-150K.NET | APIs | AWS | Angular | AzureContractor hours flexibility | Remote workMid-level ContractRemote R3d ago
-
Featured Feat. Senior Software Engineer USD 80K-170KAlgorithms | C++ | Data Structures | Go | JavaContractor flexibility | Remote workSenior-level ContractRemote R3d ago
-
Data Engineer-Secret Clearance Required USD 100K-127KAWS | AWS Glue | AWS Redshift | Azure | Azure Data401k match | Bereavement leave | Disability insurance | Employee assistance program | Employee discount programSenior-level Full TimeRemote - Nationwide, United States R9h ago
-
Data Engineer - Costa Rica USD 100K-120KAWS | Azure | Cloud Computing | Data Lake | Data WarehouseDental insurance | Disability insurance | Employee assistance program | Employee discount program | Health insuranceMid-level Full TimeCosta Rica R9h ago
-
A/B Testing | AI SDK | AWS | Agent Orchestration | B testingAnnual leave | Career growth opportunities | Company-provided laptop | Public holidays | Remote workSenior-level Full TimeRemote R12h ago
-
Data Engineer USD 115K-162KAWS | Apache Spark | Azure | Cloud platform | Data VisualizationRemote workMid-level Full TimeRemote R12h ago
-
AI Developer USD 140K-170KAPI Development | Algorithms | Amazon Web Services | Azure | CI/CDAdvanced English communication support | Remote positionMid-level Full TimeRemote R12h ago
-
Senior Data Engineer - Azure GBP 60K-70KApache Spark | Azure Data | Azure Data Factory | Azure Synapse | Azure Synapse AnalyticsRemote workSenior-level Full TimeRemote R13h ago
-
Data Scientist GenAI USD 111K-123KApache Spark | Cloud Computing | Data Analysis | Data Processing | Data cleaningComprehensive healthcare | Fully remote | International projects | Long-term contract | Multinational environmentMid-level Full TimeRemote R13h ago
-
IT Data Engineer USD 60K-65KANSI X12 | AWS | Azure | Azure Data | Azure Data Factory401k retirement plan | Disability coverage | Employee assistance program | Flexible spending account | Medical/Dental/Vision insuranceMid-level Full TimeUS - Remote R13h ago
-
Cloud Engineer USD 128K-298KAWS | Active Directory | Azure DevOps | Azure Machine Learning | Azure OpenAISenior-level Full TimeFlexible Hybrid R13h ago
-
Sr. AI Engineer USD 150K-175KAccess Control | Agentic AI | Auditability | CI/CD | Cloud platform401k | Dental insurance | Expense reimbursement for internet costs | Life insurance | Medical insuranceSenior-level Full TimeRemote, USA, United States R14h ago
-
SQL,GCP/Consultant Specialist INR 500K-1500KAWS | Agile Scrum | Backup and Recovery | Cloud Computing | Cloud Platform (GCPMid-level Full TimePune, Maharashtra, India R15h ago
-
Analytics architecture | Apache Airflow | Automation | BI Administration | Business IntelligenceCSE support | Career development | Flexible work options | Healthcare insurance subsidy | Meal ticketsSenior-level Full TimeIssy-les-Moulineaux, IDF, France R16h ago
-
Middle Data Engineer (Azure Databricks) PLN 276K-276KApache Spark | Azure Data | Azure Data Factory | Azure Databricks | Azure DevOpsHealth insurance | Mentorship | Professional development | Relocation program | Work from anywhereMid-level Full TimeWarsaw, Masovian Voivodeship, Poland R16h ago
-
Senior Data Engineer GBP 72K-80KAWS | AWS Glue | AWS Lambda | Amazon Athena | Amazon KinesisCareer progression | Flexible work-life balance | Inclusive culture | Open holiday policy | Professional developmentSenior-level Full TimeHanoi, Vietnam R16h ago
-
Mid-level Full TimeBangalore, India Office (BANGALORE) R20h ago
-
Senior Software Developer (C++) - StorageGrid INR 2000K-3500KAWS S3 | Agile | C++ | CQL | CassandraSenior-level Full TimeBangalore, India Office (BANGALORE) R20h ago
-
Security Engineer/Backend developer (StorageGRID) INR 2000K-4000KABAC | API Security | Access Management | CI/CD | Code reviewSenior-level Full TimeBangalore, India Office (BANGALORE) R21h ago
-
Senior Software Engineer (RAG Backend Developer) CNY 120K-180KA/B | A/B Testing | ABAC | Audit Logging | B testingSenior-level Full TimeGuangzhou, Guangdong, China R22h ago
-
Senior Marketing Analytics Engineer - Remote AUD 110K-120KBusiness Intelligence | Data Catalog | Data Modeling | Data Visualization | ForecastingEmployee stock options | Learning and development | Parental leave | Remote work | WFH office expense budgetSenior-level Full TimeSydney, New South Wales 2000, Australia R23h ago
-
Sr. AI Engineer - 2.00 pm to 11.00 pm IST (Remote, IND) INR 2475K-3584KAWS Bedrock | Agentic AI | Apex | Autogen | CI/CDEmployee networks | Paid adoption leave | Paid parental leave | Paid vacation and holidays | Professional development opportunitiesSenior-level Full TimeIND MH Remote, India R1d ago
-
Analyst, Data Analytics INR 1400K-2500KAI-enabled | AI-enabled analytics | Advanced Analytics | Analytics engineering | AzureComprehensive healthcare | Flexible time off | Hybrid work model | Retirement plan | Support for working parentsMid-level Full TimeMU8-South (A) Wing, 7-10 Floor, Nesco … R1d ago
-
MLOps Engineer GBP 76K-100KAirflow | Automated retraining | Autoscaling | BigQuery | Blue-Green DeploymentDiscounts | Enhanced parental leave | Holiday allowance | Learning and development programme | Life insuranceSenior-level Full TimeUnited Kingdom - Remote R1d ago
-
Edge AI Engineer USD 100K-150KBenchmarking | Bias Evaluation | C++ | Core ML | Data PrivacyCareer growth | Health benefits | Remote workSenior-level Full TimeUnited States - Remote R1d ago