Lead Data Engineer
Tasks
- Automate routine data operations and maintenance
- Build data observability solutions
- Build reusable pipeline components
- Build schedule data workflows with Apache Airflow
- Collaborate enforce data governance standards
- Conduct performance testing and implement improvements
- Create dependencies manage task execution
- Create/maintain data models
- Create/maintain documentation
- Design build maintain scalable ETL/ELT pipelines
- Design database schemas for analytics
- Develop automated data ingestion into data warehouse
- Establish data lineage tracking and documentation
- Implement DBT transformations and testing
- Implement data transformation workflows
- Implement data validation rules quality checks
- Implement error handling, retry logic, alerting
- Monitor resource utilization and cost optimization
- Monitor troubleshoot pipeline issues
- Optimize data storage and partitioning
- Tune pipelines and queries for performance and reliability
- Use Python for data processing and automation
- Work with Amazon Athena for large scale analysis
- Write optimize SQL queries
Perks/Benefits
- Active sports events and social committees
- Additional day off for birthday
- Continuous mentoring
- Employee Assistance Program 24 7 support
- Employee share plan
- Flexible arrangements
- Generous holidays
- Professional accreditations support
Skills/Tech-stack
Airbyte | Amazon Athena | Amazon Web Services | Apache Airflow | CI/CD | DBT | Data Catalog | Data Governance | Data Lineage | Data Modeling | Data Observability | Data Quality | Data Validation | Docker | ELT | ETL | Git | Infrastructure as Code | Kafka | Kinesis | Metadata Management | Partitioning | Performance Tuning | Power BI | Python | SQL | Web Services | “as-code”
Education
N/A
Roles
Data Engineer | Engineer | Lead | Lead Data | Lead Data Engineer
Related jobs
-
Lead Consultant - RAG Engineer INR 2535K-4225KAWS | AWS Step Functions | Airflow | Amazon SageMaker | Cloud NativeContinuous learning opportunities | Hands-on experience | MentorshipSenior-level Full TimeIndia-Gurugram19h ago
-
Software Engineer (Python, AI/ML) ID47296 INR 2535K-4225KCI/CD | Data Modeling | Git | Microservices | PytestBudget for education | Budget for personal growth | Budget for wellness | Local team meetups | Remote workSenior-level Full TimeNoida, India19h ago
-
Python Developer INR 2500K-3000KAWS | Debugging | Django | Event Driven | Event-driven architectureMid-level Full TimeIN-KA-Bangalore19h ago
-
Senior-level Full TimeIN-TN-Chennai19h ago
-
Databrick Developer INR 2000K-2040KAgile | Amazon Web Services | Apache Airflow | Apache Kafka | Apache SparkSenior-level Full TimeIN-TN-Chennai19h ago
-
GEN AI developer INR 2500K-3000KAPI Integration | Anthropic Claude | Context Management | Crew AI | FastAPIMid-level Full TimeIN-KA-Bangalore19h ago
-
Senior-level Full TimeIN-AP-Hyderabad20h ago
-
Azure | Csharp | Data Ingestion | Data cleaning | Machine LearningCommunity forums | Laptop provided | Medical insurance | Mentorship | No weekend workSenior-level Full TimeIndia R20h ago
-
Senior-level Full TimeGurugram, Haryana, India20h ago
-
002ASR - React Native Engineer INR 1800K-2500KAWS | Adobe XD | CI/CD | Context API | ExpoContract employment | Remote work | UK shiftSenior-level ContractTamil Nadu, Coimbatore, India21h ago
-
Software Engineer - Product (Technical Leadership) INR 2829K-5000KAI | Data Analysis | Java | JavaScript | Machine LearningSenior-level Full TimeBangalore, India21h ago
-
Senior-level Full TimeBangalore, India | Mumbai, India21h ago
-
Consultant- Data Support INR 316K-500KAlteryx | Azure Data | Azure Data Engineering | Azure Data Lake | Azure Databricks24/7 rotation | Customer service focusEntry-level Full TimeBangalore, Karnataka, India21h ago
-
Embedded C++ developer with RTOS skills INR 1000K-2000KBitbucket | C plus plus | CAN | CANopen | Change ManagementSenior-level Full Timebengaluru, India21h ago
-
Embedded C++ developer with Yocto skills INR 2000K-3200KBitbucket | C plus plus | CAN | CANopen | CI/CDSenior-level Full Timebengaluru, India21h ago
-
Microsoft Fabric Data Engineer INR 1800K-2040KAzure Data | Azure Data Factory | Azure DevOps | CI/CD | Data FactorySenior-level Full TimeThane, Maharashtra, India21h ago
-
Data Science – (Gen AI Developer) - Associate INR 2000K-2400KAutogen | CI/CD | Data Analysis | Data Visualization | ETLSenior-level Full TimeBengaluru, Karnataka, India22h ago
-
Consultant _ADF INR 2000K-2000KADF Mapping Data Flows | ARM Templates | Agile | Azure Data | Azure Data FactorySenior-level Full TimeBangalore, Karnataka, India22h ago
-
Lead Data Engineer INR 1200K-2000KAzure | Azure Blob | Azure Blob Storage | Azure Data | Azure Data FactoryUK shift workSenior-level Full TimeBengaluru, KA, India23h ago
-
India:Service Line:Consultant:Systems Support Engineer INR 1400K-1800KAWS | Agile | Azure | Azure Pipelines | CI/CD24x7 rotationMid-level Full TimeMumbai, India1d ago
-
Mid-level Full TimeMumbai, Maharashtra, India1d ago
-
Azure Data Engineer Consultant INR 1600K-3000KAzure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake Storage | Azure DatabricksDiscounts on products and services | Maternity leave | Networking opportunities | Paternity leaveMid-level Full TimeBangalore, Karnataka, India1d ago
-
Data Quality Engineer-Azure Data Platform Testing INR 1500K-2000KAgile | Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake StorageMid-level Full TimeGurugram, Haryana, India1d ago
-
Analytics Lead - Innovation Practice INR 2516K-3356KApplied statistics | Automation | Cloud Data | Cloud Data Environments | Data AnalysisSenior-level Full TimeBangalore, Karnataka, India; Pune, Maharashtra, India1d ago
-
Senior Machine Learning Engineer INR 3500K-5000KAWS | Azure | CI/CD | Cloud platform | DatabricksHybrid work | Learning and development programs | Mentorship | Remote workSenior-level Full TimeBangalore, India R1d ago