IT engineer data lakehouse
Bengaluru, KA, India
INR 1500K-2500K (estimate) Senior-level Full Time
Tasks
- Align designs with governance and metadata standards
- Apply master data management strategies
- Build bronze silver gold lakehouse layers
- Create curated data marts for reporting
- Define data quality rules and reporting
- Design scalable batch and streaming data pipelines
- Develop automated data validation tests
- Develop ingestion from structured and semi structured sources
- Document pipeline logic and data contracts
- Implement dimensional models star schema
- Implement domain driven 3NF data warehouse models
- Implement infrastructure as code with version control and CI/CD
- Monitor pipeline health and data quality
- Optimize cluster resource usage
- Profile and tune data transformation performance
- Refactor legacy pipelines for scalability
- Track data lineage and audit trails
Perks/Benefits
Skills/Tech-stack
3NF | APIs | Audit trails | Azure | Azure Databricks | Batch Processing | CI | CI Pipelines | CI/CD | Cluster optimization | Data Contracts | Data Governance | Data Ingestion | Data Lineage | Data Management | Data Modeling | Data Pipelines | Data Quality | Data Quality Rules | Data Validation | Dimensional Modeling | Domain-Driven Design | Infrastructure as Code | Integration Testing | Lakehouse | Master Data | Master Data Management | Performance Tuning | Power BI | Power Query | PySpark | Quality Rules | SAP | Scala | Slowly Changing Dimensions | Snowflake Schema | Star Schema | Streaming processing | Test-Driven | Test-Driven Development | Unit Testing | Unity Catalog | Version control | “as-code”
Education
Bachelor of Engineering | Bachelor of Science | Master of Science
Related jobs
-
Senior-level Full TimeBangalore, India Office (BANGALORE) R1d ago
-
Associate Engineer-AI/ML INR 2165K-3000KAPI Integration | Amazon Web Services | Azure | C++ | CD pipelinesEmployee discounts | Training opportunities | Tuition reimbursement | Wellness benefitsMid-level Full TimeBangalore - Karnataka, India - Kalyani …1d ago
-
Junior BI Developer/Data Engineer - India Home Base INR 300K-300KBig Data | Data Modeling | Data Pipelines | ETL | JavaFlexible remote work | Professional development opportunitiesEntry-level Full TimeBangalore, India1d ago
-
Access Control | Anomaly Detection | Audit Logging | CI/CD | ClassificationRemote working modelSenior-level Full TimeIndia1d ago
-
AI safety & Security controls, LLM Guardrails/Associate Director, Software Engineering Specialist INR 1400K-2400KAPI Development | Asynchronous processing | Audit trails | Cloud Computing | Content SafetyCareer development | Flexible working | Inclusive cultureMid-level Full TimePune, Maharashtra, India R1d ago
-
Databricks Data Engineer - Contract-to-Hire INR 1500K-3000KAWS | Apache Spark | Azure | Data Reconciliation | Data ValidationMedical insurance | Paid time off | Professional development trainingSenior-level Full TimePune, MH, India1d ago
-
Senior Databricks Data Engineer INR 1500K-3500KAWS | CI/CD | Databricks | Delta Lake | LakehouseMedical insurance | Paid time off | Performance incentives | Professional development trainingSenior-level Full TimePune, MH, India1d ago
-
Lead AI Specialist (Finance Operations) INR 3264K-5449KAccess Control | Agent Orchestration | Audit Logging | BigQuery | Change ManagementCollaborative culture | Growth opportunities | Work-life balance programsSenior-level Full TimeMumbai, MH, India1d ago
-
Senior-level Full TimeBengaluru, Karnataka, India R1d ago
-
Applied AI ML Associate INR 1400K-2400KAutomation | Data Quality | Data analytics | Data pipeline | DatabaseMid-level Full TimeBengaluru, Karnataka, India1d ago
-
AES - DE - Generative AI Solution Architect INR 2500K-4800KAI Agents | Azure | Conversational AI | Copilot Studio | Language ModelsSenior-level Full TimePune, Maharashtra, India1d ago
-
Lead Assistant Manager INR 1500K-2500KApache Airflow | BigQuery | Cloud Composer | Cloud Storage | Cloud platformSenior-level Full TimePune, Maharashtra, India1d ago
-
Mid-level Full TimeBangalore, Karnataka, India R1d ago
-
AI Architect / Sr. AI Engineer INR 2000K-5000KAnthropic Claude | CI/CD | Chroma | Django | Django RESTRemote workSenior-level Full TimePune R1d ago
-
Senior Software Engineer(AI) INR 2500K-4580KAPI Development | Access Control | Agent Builder | Alerts | Automated testingBirthday off | Flexible working hoursSenior-level Full TimeGurgaon1d ago
-
Data Engineer Consultant Senior INR 3000K-4000KAgile | Alerting | Auditability | Automated testing | AzureBike to work | Career mentorship | Educational assistance | Employee assistance program | Employee discountSenior-level Full TimeBengaluru, KA, India R1d ago
-
Data Engineer Consultant Principal INR 3000K-4000KAccess Control | Agile | Audit and compliance | Auditability | Automated testingBike to work | Career progression mentorship | Educational assistance | Employee assistance programme | Employee discount schemeSenior-level Full TimeBengaluru, KA, India R1d ago
-
API Testing | CI/CD | Postman | Pytest | RESTCompany-paid certifications | Language courses | Medical insurance | Mental health support | Udemy accessSenior-level Full TimeIndia1d ago
-
AWS | Agile | Apache Spark | Azure | CI/CDEducation and learning support | Generous time off | Professional growth | Remote work | Wellness and lifestyle resourcesSenior-level Full TimeIndia R1d ago
-
Senior Engineer- Fullstack INR 2500K-5140KAPIs | Access Control | Backend Services | Data Access Control | Data ModelingHybrid work model | In-office collaborationSenior-level Full TimeBangalore, Karnataka, India R1d ago
-
Senior ML Operations Engineer INR 2500K-5000KAI Governance | AWS | Automated retraining | Azure | Azure MLSenior-level Full TimeBengaluru, Karnataka, India1d ago
-
SR PROGRAMMER 10514- Salesforce Platform | Lightning Web Components (LWC) | Agentforce | CRM Analytics (CRMA) | Apex | Flow INR 1800K-2800KAgentforce | Apex | CI/CD | CRM Analytics | CSSSenior-level Full TimeBangalore1d ago
-
API Integration | Agent Orchestration | CI/CD | Change Management | Confidence scoringFlexible working | Inclusive work environment | Professional development opportunitiesMid-level Full TimeHyderabad, Telangana, India R1d ago
-
AI Engineer INR 3000K-5000KAI Services | AWS SageMaker | Azure AI | Azure AI Services | Azure DevOpsMid-level Full TimeIndia1d ago
-
Software Development Specialist INR 940K-940KAmazon CloudWatch | Amazon DynamoDB | Amazon S3 | Amazon Web Services | Azure DevOpsMid-level Full TimeBangalore, KA, IN R1d ago