Senior Data Engineer - Senior Manager
Tasks
- Automate deployment of Databricks jobs notebooks and configurations
- Build CI CD pipelines for data workflows and infrastructure
- Build batch and streaming ETL/ELT pipelines
- Design data ingestion and transformation pipelines
- Develop data quality checks validation rules and reconciliation
- Implement Apache Iceberg schema evolution partitioning and compaction
- Implement data governance access control and metadata management
- Implement lakehouse architecture and manage bronze silver gold layers
- Implement observability, monitoring, and alerting
- Manage job orchestration workflows scheduling retries and failure recovery
- Monitor end to end ingestion pipelines
- Optimize Spark jobs for cost performance and scalability
Perks/Benefits
- Employee networks
- Flexible work/life support
- Inclusive development opportunities
- Paid volunteer days
Skills/Tech-stack
AWS | Apache Airflow | Apache Iceberg | Apache Spark | CI/CD | Data Governance | Data Quality | Data Reconciliation | Data Validation | Databricks | Databricks Jobs | ELT | ETL | Git | Lakehouse | Observability | PySpark | RBAC | SQL | Terraform | Unity Catalog | Workflow Orchestration
Education
N/A
Related jobs
-
Mid-level Full TimeHaveli, India2h ago
-
Senior AI Architect INR 2000K-5000K.NET | AI Foundry | AI Governance | AI Security | API IntegrationSenior-level Full TimeChennai, India3h ago
-
Senior-level Full TimeGurugram, Haryana, India4h ago
-
Senior-level Full TimeIN-TN-Chennai4h ago
-
Senior-level Full TimeIN-AP-Hyderabad4h ago
-
Senior-level Full TimeIN-AP-Hyderabad4h ago
-
Senior Artificial Intelligence/Machine Learning Engineer INR 3000K-5000KAPI Integrations | Agentic AI | Alerting | Atlassian Bamboo | Atlassian BitbucketCompany-paid certifications | Financial consultations | Language courses | Legal consultations | Medical insuranceSenior-level Full TimeIndia5h ago
-
Staff Engineer INR 4200K-6944KAgentic Workflows | Anomaly Detection | Apache Kafka | BigQuery | Cloud platformSenior-level Full TimeChennai, Tamil Nadu, India5h ago
-
Associate - Data Scientist INR 1500K-2500KAmazon Web Services | Apache Spark | Artificial neural networks | Chunking | DatabricksMid-level Full TimeGurugram, Haryana, India6h ago
-
Partner Forward Deployed Engineer, GenAI, Google Cloud INR 2500K-5000KAgent systems | Agent workflows | Application Programming | Application Programming Interfaces | Computer VisionSenior-level Full TimeMumbai, Maharashtra, India6h ago
-
Assistant Manager INR 1500K-2200KACID | Azure Cloud | Azure Data | Azure Data Factory | CI/CDWork from office 5 daysMid-level Full TimeBengaluru, Karnataka, India6h ago
-
Azure Data Engineer INR 1500K-2500KAzure Cloud | Azure Cloud Platform | Azure Data | Azure Data Factory | Azure Data LakeSenior-level Full TimeHyderabad, Telangana, India6h ago
-
Senior Software Engineer _Cloud Developer INR 3715K-5000KAPI Development | API Integration | AWS Glue | AWS Lambda | AWS Step FunctionsEqual opportunity | Inclusive culture | On-call rotationSenior-level Full TimeHyderabad, India7h ago
-
Senior Artificial Intelligence/Machine Learning Engineer INR 3000K-5000KAPIs | Agentic AI | Apache Flink | Apache Spark | CI/CDCompany-paid certifications | Financial consultations | Language courses | Legal consultations | Medical insuranceSenior-level Full TimeIndia7h ago
-
Data & Analytics Engineer - AM INR 1500K-2700KApache Spark | Data Modeling | Data Warehousing | Databricks | Delta LakeSenior-level Full TimeBangalore, Karnataka, India7h ago
-
Senior Artificial Intelligence/Machine Learning Engineer INR 2500K-4800KAWS | Agent Frameworks | Agent systems | Autogen | CI/CDCompany-paid certifications | Financial consultations | Language courses | Legal consultations | Medical insuranceSenior-level Full TimeIndia7h ago
-
Lead Software Engineer Java INR 2500K-4000KAPI Gateway | AWS Glue | AWS Lambda | AWS OpenSearch | AWS SageMakerSenior-level Full TimeBengaluru, Karnataka, India8h ago
-
Data and Analytics Engineer , Enablement INR 3000K-4000KApache Spark | Azure Data | Azure Data Platform | CI/CD | Data AnalysisDocumentation enablement content creation | Mentoring and onboarding supportSenior-level Full TimeMumbai, Maharashtra, India8h ago
-
Data and Analytics Engineer , Enablement INR 3000K-4000KAgile | Apache Spark | Data Governance | Data Modeling | Data VisualizationSenior-level Full TimeMumbai, Maharashtra, India8h ago
-
Data and Analytics Engineer , Enablement INR 3000K-4000KAgile | Apache Spark | Azure Data | Azure Data Platform | CI/CDSenior-level Full TimeMumbai, Maharashtra, India8h ago
-
Data and Analytics Engineer , Enablement INR 3000K-4000KCI/CD | Data Analysis | Data Modeling | Databricks | Exploratory Data AnalysisMid-level Full TimeMumbai, Maharashtra, India8h ago
-
Senior Software Engineer INR 3264K-5000KAPI Testing | AWS | Amazon Web Services | Ansible | Artificial IntelligenceFamily wellbeing benefits | Health and wellbeing benefits | Paid time offSenior-level Full TimeHyderabad, India9h ago
-
Software Engineer III - Data Lake INR 2000K-3100KAWS | AWS Glue | Agile methodologies | Amazon S3 | Apache CassandraSenior-level Full TimeHyderabad, Telangana, India10h ago
-
Senior Software Engineer INR 3264K-4720KAI Engineering | Best practices | Business Intelligence | Cloud Computing | Data ArchitectureFamily support benefits | Flexible working options | Health benefits | Paid time off | Retirement benefitsSenior-level Full TimeIndia10h ago
-
Staff Software Engineer - Database Engineering INR 2771K-4064KAmazon Aurora | Ansible | Cassandra | Citus | Configuration ManagementSenior-level Full TimeBangalore, India R10h ago