Senior Databricks Engineer
Chennai, Tamil Nadu, India
INR 1500K-2500K (estimate) Senior-level Full Time
Tasks
- Build Delta Live Tables pipelines
- Build and optimize Databricks pipelines
- Configure Unity Catalog governance and access control
- Design and implement Delta Lake tables
- Design data lakehouse solutions on Databricks
- Develop batch and streaming pipelines with PySpark and Spark SQL
- Implement error handling retry logic and dead letter queue patterns
- Implement medallion architecture bronze/silver/gold
- Ingest real time data from Kafka Event Hubs and Kinesis into Delta
- Maintain Databricks workspaces clusters and compute pools
- Manage Databricks jobs workflows and orchestration
- Manage schema evolution table versioning time travel and change data feed
- Optimize cluster configurations for cost and performance
Perks/Benefits
- N/A
Skills/Tech-stack
Access Control | Adaptive Query | Adaptive query execution | Amazon Kinesis | Amazon S3 | Apache Kafka | Auto Scaling | Azure Data | Azure Data Lake | Azure Data Lake Storage | Azure Event | Azure Event Hubs | Broadcast Join | Change Data Feed | Cloud Storage | Cluster auto-scaling | Data Governance | Data Lake Storage | Data Lineage | Data Partitioning | Data feed | Databricks | Delta Lake | Delta Live Tables | Delta Live) | Event Hubs | Google Cloud | Google Cloud Storage | Medallion Architecture | PySpark | Query Execution | Schema evolution | Spark SQL | Spot instances | Time Travel | Unity Catalog | Z-ordering
Education
Related jobs
-
Lead AI Engineer ID80141 INR 2500K-5000KAWS S3 | Amazon Web Services | Azure | CI/CD | Cloud SecurityAnnual learning budget | Collaborative culture | Flexible hours | Internal TechTalks | MentorshipSenior-level Full TimeBengaluru, India4h ago
-
Staff Data Engineer INR 1500K-2500KAPIs | AWS | Agentic Workflows | Automated testing | Cloud PlatformsSenior-level Full TimeHyderabad, Telangana , India R6h ago
-
Data Engineer, YouTube Business Organization INR 1000K-2000KApache Flume | Apache Spark | Data Governance | Data Lakes | Data ModelingMid-level Full TimeBengaluru, Karnataka, India6h ago
-
Sr Advanced Data Engineer INR 1500K-3000KAWS | Apache Spark | Azure | Data Cleansing | Data ModelingSenior-level Full TimeBengaluru, Karnataka, India6h ago
-
Senior Data Engineer INR 3000K-4000KAWS | Airflow | Apache Spark | CI/CD | DBTBike to work | Career mentorship | Educational assistance | Employee assistance programme | Employee discountSenior-level Full TimeMumbai, MH, India R8h ago
-
Data Engineer 2 INR 1500K-2500KAgile Development | Cassandra | Cloud Computing | Data Catalog | Data GovernanceSenior-level Full TimePune, Maharashtra, India8h ago
-
Data Engineer Manager - AWS Databricks INR 1500K-2600KAWS | Apache Spark | Automation | CI/CD | Data GovernanceSenior-level Full TimeHyderabad, TS, India8h ago
-
Data Engineer INR 3000K-4000KAmazon S3 | Databricks | ELT | ETL | PythonBike to work | Educational assistance | Employee assistance program | Employee discount | Enhanced parental leaveSenior-level Full TimeBengaluru, KA, India R9h ago
-
Senior Software Engineer INR 1163K-3000KAutomated testing | Batch Processing | CI/CD | DAX | Data GovernanceFlexible work arrangement | Hybrid work modelSenior-level Full TimeIndia, Telangana, Hyderabad R9h ago
-
Senior / Lead Data Engineer - Azure Databricks INR 1500K-3000KAzure Databricks | Azure DevOps | CI/CD | Cloud Security | Conformed DimensionsSenior-level Full TimeHyderabad, TS, India9h ago
-
Lead Software Engineer - Java/Python, Data Engineering INR 3000K-4200KAPI Design | AWS | Airflow | Apache Iceberg | AzureSenior-level Full TimeHyderabad, Telangana, India10h ago
-
Lead Software Engineer - Java/Python, Data Engineering INR 3000K-4200KAPI Design | AWS | AWS Bedrock | Airflow | Apache IcebergSenior-level Full TimeHyderabad, Telangana, India10h ago
-
Lead Software Engineer - Java/Python, Data Engineering INR 3000K-4200KAPI Design | AWS | AWS Bedrock | Apache Airflow | Apache IcebergSenior-level Full TimeHyderabad, Telangana, India10h ago
-
Sr Lead Software Engineer - Java/Python, Data Engineering INR 2500K-4300KAPI Design | AWS | Airflow | Apache Iceberg | Apache KafkaSenior-level Full TimeHyderabad, Telangana, India10h ago
-
Solution Database Engineer INR 1500K-2600KAgile | Azure Data | Azure Data Factory | Azure DevOps | Common table expressionsSenior-level Full TimeBengaluru, KA, India R12h ago
-
Lead Associate - SQL Developer INR 800K-1500KAzure Data | Azure Data Factory | BI | Big Data | Data ArchitectureMid-level Full TimePune13h ago
-
Lead Associate - Data / AI INR 1500K-2500KAzure Data | Azure Data Factory | Big Data | Cause analysis | Data FactoryMid-level Full TimePune13h ago
-
Azure Databricks_Technical Architect INR 2500K-4500KAgile methodologies | Azure Databricks | Cloud Data | Cloud Data Platforms | Data EngineeringHolidays | Paid time off | Rewards and recognition programs | Work-life balanceSenior-level Full TimeHyderabad, TELANGANA, India14h ago
-
IN|KDN KRC|Nordics|Digital Audit -Tech Senior - BLR INR 2400K-4000KAlteryx | Data Governance | Data Hygiene | Data Modeling | Data ValidationSenior-level Full TimeBangalore, Karnataka, India15h ago
-
AWS Bedrock | AWS Lambda | Amazon Redshift | CI/CD | Context OptimizationEmployee recognition incentives | Generous leave | Health insurance | Immigration Support Programs | Professional growth trainingMid-level Contract Full TimeIndia R15h ago
-
AWS | AWS Lambda | Access Control | Access History | Access ManagementAugust Free Fridays | Education support | Employee Well Being Resources | Flexible work options | Leadership development programsSenior-level Full TimeIndia15h ago
-
Mid-level Full TimeIndia New Delhi (Noida)18h ago
-
Senior Cloud Engineer INR 1500K-4000KAPI Integration | ARM Templates | AWS Glue | AWS Lambda | Access ManagementSenior-level Full TimeIndia New Delhi (Noida)18h ago
-
FBS Associate Analytics Engineer INR 1000K-2000KAWS | Apache Airflow | Azure | Cloud platform | DBTCareer development | Flexible work arrangements | Paid time off | Pension plan | Private health insuranceMid-level Full TimeHyderabad, Telangana, India R18h ago
-
API | AWS CloudFormation | AWS CloudTrail | AWS Glue | AWS Glue Data CatalogCareer development opportunities | Coaching and feedback | Flexible work approachSenior-level Full TimeHyderabad, TG, IN, 50008118h ago