Sr Data Engineer, Python + Spark (Data Federation skillset - Data Lakehouse - Eg: Starburst) - Chennai
Tasks
- Build ETL/ELT pipelines
- Design analytics data models
- Design data federation architecture
- Develop distributed data processing jobs
- Implement data lakehouse
- Implement data masking
- Implement federated query layer
- Implement fine grained access control
- Manage Delta lake
- Optimize Apache Spark jobs
- Tune SQL query performance
Perks/Benefits
- 401k retirement plan
- Dental insurance
- Medical insurance
- Paid Holidays
- Paid time off
- Variable pay/incentives
- Vision insurance
Skills/Tech-stack
AWS EMR | AWS Glue | AWS S3 | Apache Hudi | Apache Iceberg | Apache Spark | Apache Trino | Azure Data | Azure Data Lake | Azure Data Lake Storage | Azure Databricks | Cloud platform | DBT | Data Lake Storage | Data Modeling | Data fabric | Data mesh | Delta Lake | Dremio | Google Cloud | Google Cloud Platform | Infrastructure as Code | Kubernetes | Medallion Architecture | Microsoft Azure | Microsoft Azure Databricks | Pulumi | PySpark | Python | SQL | Starburst | Terraform | “as-code”
Education
N/A
Roles
Related jobs
-
Senior Data Engineer- AWS INR 2000K-2500KAWS Glue | AWS Lambda | Amazon Redshift | Amazon Web Services | Apache AirflowSenior-level Full TimeBangalore North, India9h ago
-
Java, GCP, ReactJS/Consultant Specialist INR 2200K-3200KAnsible | Apache Airflow | Apache Atlas | BigQuery | ClaudeFlexible working | Professional developmentMid-level Full TimeHyderabad, Telangana, India R11h ago
-
Senior-level Full TimeIN-KA-Bangalore11h ago
-
Java, GCP, ReactJS/Consultant Specialist INR 2200K-3200KAnsible | Apache Airflow | Apache Atlas | BigQuery | ClaudeFlexible working | Professional developmentMid-level Full TimeHyderabad, Telangana, India R11h ago
-
Consultant - Cloud AI Developer INR 1400K-2500KAWS | Application Modernization | Application rationalization | Azure | Azure OpenAIMid-level Full TimeBangalore, Karnataka, India12h ago
-
AI Outcome Customer Engineer, FDE, Google Cloud INR 3799K-6000KAI Model Deployment | AI model | APIs | Access Management | Cloud infrastructureSenior-level Full TimeMumbai, Maharashtra, India; Bengaluru, Karnataka, India12h ago
-
AI Customer Engineer, Developer AI, Google Cloud INR 2500K-5000KAI Agents | API Gateway | Audit Logging | Cloud platform | Distributed tracingConference participation | Travel opportunitiesSenior-level Full TimeBengaluru, Karnataka, India12h ago
-
Mid-level Full TimePune, Maharashtra, India13h ago
-
DTICI_SalesProductServices_DataEngineer_T9 INR 1500K-2500KAWS | Agile | Apache Spark | Azure | Azure DataMid-level Full TimeBengaluru, Karnataka, India13h ago
-
Entry-level Full TimePune, Maharashtra, India13h ago
-
AI Full stack Engineer INR 2000K-5000KAPI Development | Agentic AI | BigQuery | Cloud SQL | Cloud platformSenior-level Full TimeIndia13h ago
-
Associate Process Manager INR 500K-1500KAzure SQL | Azure Synapse | BI Data modeling | Compliance | DAXMid-level Full TimeMumbai, Maharashtra, India13h ago
-
Agentic AI | Automation Anywhere | Azure OpenAI | Databases | Generative AIEntry-level Full Timebangalore, India14h ago
-
Lead Software Engineer - Data Platforms, REST APIs, SQL INR 3000K-5000KAWS | Agile | Authentication | Automated testing | CI/CDSenior-level Full TimeHyderabad, Telangana, India14h ago
-
QA (ETL Test Engineer) INR 1000K-2000KAgile Scrum | Azure DevOps | Batch Processing | Data Modeling | Data ProcessingMid-level Full TimeNoida, Uttar Pradesh, India15h ago
-
Senior Analyst - AI/ML INR 2500K-5000KAWS | Artificial Intelligence | Automl | Azure | Cloud platformMentorship | TrainingSenior-level Full TimePune, Maharashtra, IND, 41101416h ago
-
Principal Software Engineer INR 1500K-3000KApache Spark | C# | C++ | Continuous integration | Distributed SystemsSenior-level Full TimeIndia, Karnataka, Bangalore17h ago
-
Mid-level Full TimeBengaluru, Karnataka, India17h ago
-
Data Engineering / Sr. Associate Director, Software Engineering INR 1000K-2000KAPIs | Airflow | Apache Beam | Auditability | Backwards CompatibilityCareer developmentMid-level Full TimeHyderabad, Telangana, India R17h ago
-
Mid-level Full TimeBengaluru, Karnataka, India17h ago
-
Mid-level Full TimeHadapsar, Pune, Maharashtra, IND, 41101317h ago
-
AI Data Scientist INR 1200K-3000KAgent systems | Agentic AI | Data Preprocessing | Data Quality | Diffusion ModelsMid-level Full TimeChennai, Tamil Nadu, India18h ago
-
Software Engineer Data Analytics INR 1687K-2500KAPI Development | Agile | AppSheet | Apps Script | AutomationCollaborative global team environment | Flexible work modelMid-level Full TimeHyderabad, India18h ago
-
Data Engineer INR 360K-480KApache Airflow | Apache Flink | Git | Linux | PandasCareer development | Code reviews | MentorshipEntry-level Full TimeMumbai, Maharashtra, India18h ago
-
Senior-level Full TimeBengaluru, Karnataka, India18h ago