Senior Data Engineer
Tasks
- Analyze pipelines and improve reliability scalability
- Build real-time data pipelines with Kafka
- Build scalable data ingestion pipelines
- Collaborate on data quality dashboards and reporting
- Design lakehouse architectures with Apache Iceberg
- Develop Apache Spark batch and streaming jobs
- Establish observability with monitoring logging and alerting
- Implement data quality checks and error monitoring
- Implement versioning time travel incremental processing and ACID compliance
- Optimize Iceberg table design
- Orchestrate workflows with DBT and Airflow
- Perform code reviews and apply engineering best practices
- Use AI tools to accelerate ETL and ELT development
- Use AWS services for cost and performance optimization
Perks/Benefits
- N/A
Skills/Tech-stack
AWS Glue | AWS Lambda | AWS Step Functions | Alerting | Amazon EMR | Amazon S3 | Amazon Web Services | Apache Airflow | Apache Iceberg | Apache Kafka | Apache Spark | CI/CD | DBT | Data Quality | Data Validation | Kafka Streams | Logging | Monitoring | Python | SQL | Snowflake | Step Functions | Web Services
Education
Roles
Related jobs
-
Algorithms | Artificial Intelligence | BigQuery | Cloud Storage | Cloud platform24x7 support rotation | Non-standard hours | On-call workMid-level Full TimeBengaluru, Karnataka, India; Pune, Maharashtra, India3h ago
-
Senior Software Engineer, Silicon Infrastructure and Engineering INR 1500K-2300KAlgorithms | Business Intelligence | Dashboards | Data Structures | Data VisualizationSenior-level Full TimeBengaluru, Karnataka, India3h ago
-
Azure Data Engineer INR 1500K-2000KAzure Data | Azure Data Factory | Azure Data Lake | Azure Functions | Azure SynapseMid-level Full TimeChennai, Tamil Nadu, India5h ago
-
SDE - Backend* INR 1600K-2200KAlgorithms | CI/CD | Data Structures | Distributed Systems | DockerCollaborative culture | Hybrid work | Leadership development programs | Learning and development | Multicultural work environmentMid-level Full TimeIndia6h ago
-
SDE - Backend* INR 1600K-2200KAlgorithms | Availability | CI | CI/CD | Code reviewCollaborative culture | Hybrid work | Learning and development programs | Multicultural experience | Professional growthMid-level Full TimeIndia6h ago
-
Lead Assistant Manager INR 1000K-2000KACID | Batch Processing | CI/CD | Data Ingestion | Data ModelingWork from office 5 days per weekSenior-level Full TimeGurugram, Haryana, India7h ago
-
Ml Engineer Ii INR 302K-420KAnomaly Detection | Classification | Clustering | Data Pipelines | Feature EngineeringEntry-level Full TimeBANGALORE, KA, IN, 5600488h ago
-
AES - DE - Generative AI Application Developers INR 2000K-5000KAWS | Application Security | Authentication | Authorization | AzureSenior-level Full TimePune, Maharashtra, India8h ago
-
AES - DE - Generative AI Application Developers INR 2800K-3500KAWS | Application Security Testing | Authentication | Authorization | AzureSenior-level Full TimePune, Maharashtra, India8h ago
-
AI Foundry | AWS | Azure | Azure AI | Azure AI FoundryFlexibility programmes | Inclusive benefits | Mentorship | Wellbeing supportMid-level Full TimeBhubaneswar - Ihub, India14h ago
-
AI Foundry | AWS SageMaker | Amazon Web Services | Azure | Azure AIFlexible work arrangements | Inclusive benefits | Mentorship | Wellbeing supportMid-level Full TimeBhubaneswar - Ihub, India14h ago
-
AI Foundry | AWS | Amazon SageMaker | Amazon SageMaker Pipelines | AnthropicFlexibility programs | Inclusive benefits | Mentorship | Wellbeing supportMid-level Full TimeBhubaneswar - Ihub, India14h ago
-
AI Foundry | Amazon SageMaker | Azure AI | Azure AI Foundry | Azure MLFlexibility programs | Inclusive benefits | Mentorship | Wellbeing supportMid-level Full TimeBhubaneswar - Ihub, India14h ago
-
AWS | Apache Airflow | Apache Hadoop | Azure Data | Azure Data FactoryFlexible work programmes | Inclusive benefits | Mentorship | Wellbeing supportSenior-level Full TimeKolkata Y-14, India14h ago
-
Amazon SageMaker | Amazon SageMaker Pipelines | Amazon Web Services | Azure | Azure Machine LearningMid-level Full TimeBhubaneswar - Ihub, India14h ago
-
Amazon Web Services | Anthropic | Azure | Azure Machine Learning | CI/CDMid-level Full TimeBhubaneswar - Ihub, India14h ago
-
Apache Airflow | Apache Hadoop | Azure Data | Azure Data Factory | Chroma DBFlexibility programs | Inclusive benefits | MentorshipSenior-level Full TimeKolkata Y-14, India14h ago
-
IN-Associate_Azure Databricks _ Data and Analytics_Advisory_Pune INR 2040K-2400KAnalytics Cloud | Azure Data | Azure Data Factory | BW/4HANA | Data FactoryFlexibility programmes | Inclusive benefits | Mentorship | Wellbeing supportSenior-level Full TimePune, India14h ago
-
IN_Associate 2_Azure Data Engineering _OC D&A_Advisory_Kolkata INR 1500K-2000KAgile | Azure | Azure Data | Azure Data Factory | Azure SQLFlexibility programs | Inclusive benefits | Mentorship | Wellbeing supportMid-level Full TimeKolkata Y-14, India14h ago
-
Agile | Azure Data | Azure Data Factory | Azure SQL | Data ComplianceFlexible work programs | Inclusive benefits | MentorshipSenior-level Full TimeKolkata Y-14, India14h ago
-
IN_Associate 2_Azure Data Engineering _OC D&A_Advisory_Kolkata INR 1500K-2000KAgile | Azure | Azure Data | Azure Data Factory | Azure SQLFlexibility programs | Inclusive benefits | MentorshipMid-level Full TimeKolkata Y-14, India14h ago
-
Lead Software Engineer - AI/ML INR 3125K-4512KAgentic Systems | CI/CD | Containerization | Data Modeling | Data analyticsRelocation assistanceSenior-level Full TimeIND - Bangalore, India14h ago
-
Lead Software Engineer - AI/ML INR 2500K-4500KC++ | CI/CD | CUDA | Calibration | Computer VisionRelocation within IndiaSenior-level Full TimeIND - Bangalore, India14h ago
-
AWS | C# | C++ | CI/CD | Cloud platformSenior-level Full TimePune - Tower 6, India R14h ago
-
AI Foundry | AWS | Amazon Web Services | Azure | Azure AIFlexibility programmes | Inclusive benefits | MentorshipMid-level Full TimeBhubaneswar - Ihub, India14h ago