Senior Databricks Engineer
Chennai, Tamil Nadu, India
INR 1500K-2500K (estimate) Senior-level Full Time
Tasks
- Build Delta Live Tables pipelines
- Build and optimize Databricks pipelines
- Configure Unity Catalog governance and access control
- Design and implement Delta Lake tables
- Design data lakehouse solutions on Databricks
- Develop batch and streaming pipelines with PySpark and Spark SQL
- Implement error handling retry logic and dead letter queue patterns
- Implement medallion architecture bronze/silver/gold
- Ingest real time data from Kafka Event Hubs and Kinesis into Delta
- Maintain Databricks workspaces clusters and compute pools
- Manage Databricks jobs workflows and orchestration
- Manage schema evolution table versioning time travel and change data feed
- Optimize cluster configurations for cost and performance
Perks/Benefits
- N/A
Skills/Tech-stack
Access Control | Adaptive Query | Adaptive query execution | Amazon Kinesis | Amazon S3 | Apache Kafka | Auto Scaling | Azure Data | Azure Data Lake | Azure Data Lake Storage | Azure Event | Azure Event Hubs | Broadcast Join | Change Data Feed | Cloud Storage | Cluster auto-scaling | Data Governance | Data Lake Storage | Data Lineage | Data Partitioning | Data feed | Databricks | Delta Lake | Delta Live Tables | Delta Live) | Event Hubs | Google Cloud | Google Cloud Storage | Medallion Architecture | PySpark | Query Execution | Schema evolution | Spark SQL | Spot instances | Time Travel | Unity Catalog | Z-ordering
Education
Related jobs
-
Agentic AI Data Engineer INR 2500K-5000KAWS Glue | AWS Lambda | AWS Step Functions | Amazon Athena | Amazon BedrockSenior-level Full TimeGurugram, Haryana, India3h ago
-
Data Scientist INR 2500K-4200KAmazon SageMaker | Azure AI | Azure Synapse | Azure Synapse Analytics | C++Mid-level Full TimeHyderabad, Telangana, India6h ago
-
Data Engineer INR 1500K-2980KAPI Security | AWS | Angular | CSS3 | CypressEmployee assistance program | Employee discounts | Gym reimbursement | Healthcare coverage | Learning platform accessMid-level Full TimeBangalore, Karnataka, India7h ago
-
Software Engineer - Tableau INR 2520K-4000KA/B | A/B Testing | Agile | Attribution Modeling | Audience SegmentationFlexible work modelSenior-level Full TimeHyderabad, India7h ago
-
Senior-level Full TimePune, MH, India8h ago
-
Data Engineer INR 2771K-4064KAgile | Azure Data | Azure Data Services | Azure DevOps | Cloud ConceptsLearning and career development | Management training | Modern infrastructure | Online learningSenior-level Full TimeChennai, IN, 600 03213h ago
-
Kafka Administrator INR 2800K-4200KAWS EC2 | AWS Glue | AWS IAM | AWS MSK | AWS SecretsBudget for courses and technical conferences | Career progression | Funded certification paths | Hybrid/Remote flexibility | MentorshipSenior-level Full TimeBengaluru, Karnataka, India17h ago
-
Senior-level Full TimeHyderabad, India22h ago
-
Senior-level Full TimeHyderabad, India1d ago
-
Data Engineer, Cloud Supply Chain Data Engineering INR 1500K-2200KBigQuery | Data Governance | Data Monitoring | Data Pipelines | Data QualityCollaborative team environment | Mentoring opportunitiesMid-level Full TimeBengaluru, Karnataka, India1d ago
-
Senior Staff Software Engineer, Geo INR 2500K-3465KAnomaly Detection | C++ | Continuous Learning | Data Governance | Data InfrastructureSenior-level Full TimeBengaluru, Karnataka, India1d ago
-
Mid-level Full TimePune, Maharashtra, India1d ago
-
Data Engineer INR 3000K-4000KData Ingestion | Data Pipelines | Data Quality | Data Transformation | Data ValidationSenior-level Full TimePune, Maharashtra, India1d ago
-
EAI Software Engineer INR 1500K-2100KAgile | Airflow Orchestration | Apache Airflow | CI/CD | Cloud ComposerMid-level Full TimeHyderabad, India1d ago
-
Senior Software Engineer (Storage & Cloud) INR 3000K-4000KAWS | Agile Software Development | Agile software | Ansible | Automated testingSenior-level Full TimeBangalore, India Office (BANGALORE)1d ago
-
Senior AI/ML Engineer I INR 2771K-4500KAI vector search | Apache Spark | Autogen | Azure DevOps | CI/CDCareer advancement programs | Certification programs | Continuous learning opportunities | Dental coverage | Education reimbursement programSenior-level Full TimeBangalore, Karnataka, India1d ago
-
Mid-level Full TimeGurugram, Haryana, India1d ago
-
AWS | AWS Lambda | Amazon EMR | Amazon S3 | Apache AirflowEmployee Assistance Program (EAP) | Flexible working environment | LinkedIn Learning | Volunteer time off | Wellness plansSenior-level Full TimePune, MH, India1d ago
-
DE&A - AIML - Data Science - Machine Learning INR 2500K-5000KAI orchestration | AWS | Agentic AI | Azure | CI/CDSenior-level Full TimeIndia1d ago
-
AWS | Agentic Workflows | Azure | CI/CD | Cloud platformCharity matching | Corporate volunteering | Dental insurance | Medical insurance | Paid time offMid-level Full TimeBengaluru, KA, IN1d ago
-
Engineering-L2-Bengaluru-Associate-Software Engineering INR 1500K-2500KApache Spark | Avro | CI/CD | Cause analysis | ClusteringMid-level Full TimeBengaluru, Karnataka, India1d ago
-
Engineering-L2-Bengaluru-Analyst-Software Engineering INR 1500K-2500KApache Spark | CI/CD | Cause analysis | Clustering | Data ModellingEntry-level Full TimeBengaluru, Karnataka, India1d ago
-
Senior-level Full TimeBengaluru1d ago
-
Data Engineer I INR 1500K-2000KAWS | Apache Spark | Automated testing | Azure | Azure DataHybrid work | Inclusive work environment | Professional developmentMid-level Full TimeBangalore, Karnataka, IN1d ago
-
Data Engineer I INR 1500K-2000KApache Spark | Automated testing | Azure Data | Azure Data Factory | Azure DatabricksCareer development | Learning opportunitiesMid-level Full TimeBangalore, Karnataka, IN1d ago