Senior Data Architect – AWS & Databricks Modernization
IN: Lilly Bengaluru, India
INR 3000K-4000K (estimate) Senior-level Full Time
Tasks
- Apply MLflow and LLM tooling for schema mapping and data quality
- Architect and engineer Databricks lakehouse using medallion design
- Automate lineage capture schema drift detection and post migration validation
- Build and optimize Delta Live Tables pipelines and Spark jobs
- Build migration pipelines and replatforming accelerators
- Build reusable governed data products and data sharing
- Define KPIs and report data driven decision metrics
- Define reusable modernization patterns and environment topology
- Define semantic modeling roadmap and analytics ready data
- Design and rollout Unity Catalog governance
- Drive AI assisted migration gap analysis and documentation
- Implement data quality rules and remediation workflows for clinical data
- Implement role based and attribute based access control and encryption
- Own lakehouse modernization roadmap and execution
- Package and deploy pipelines with Databricks Asset Bundles CI CD
- Partner with SMEs and engineers to plan and sequence migrations
- Tune performance cost using Photon cluster policies and autoscaling
Perks/Benefits
- N/A
Skills/Tech-stack
AWS | Access Control | Asset bundles | Attribute-Based Access Control | Attribute-based access | CI/CD | Cluster policies | Data Governance | Data Quality | Databricks | Databricks (asset) | Databricks Asset Bundles | Databricks SQL | Databricks Workflows | Delta Lake | Delta Live Tables | Delta Live Tables pipelines | Delta Live) | Dimensional Modeling | Drift Detection | Encryption | LLM | Lakehouse Federation | Lineage | MLflow | Mosaic AI | Ontology | Photon | PySpark | Role-Based Access | Role-Based Access Control | Schema drift | Schema drift detection | Schema modeling | Semantic layer | Spark SQL | Unity Catalog
Education
N/A
Roles
Related jobs
-
Senior-level Full TimeShaikpet, India R18h ago
-
Senior-level Full TimeNoida,Uttar Pradesh,India1d ago
-
DE-Cloud Native Data Architect-AWS-GDSN02 INR 2771K-4300KAI-powered analytics | AWS | AWS Glue | AWS Lake | AWS Lake FormationSenior-level Full TimeBengaluru, KA, IN, 5600161d ago
-
Generative AI Architect- Manager INR 2000K-5000KAI Foundry | Amazon Bedrock | Artificial Intelligence | Autogen | Azure AISenior-level Full TimeBangalore (SDC) - Bagmane Tech Park, …1d ago
-
Generative AI Architect- Senior Manager INR 2000K-5000KAI Foundry | AI Observability | AI Search | Agent communication | Agent systemsSenior-level Full TimeBangalore (SDC) - Bagmane Tech Park, …1d ago
-
Data Architecture - Advisor I INR 2515K-3900KData Architecture | Data Governance | Data Modeling | Data integration | Database DesignSenior-level Full TimeThane, India1d ago
-
Lead Engineer - Ad Tech INR 3470K-5000KAPI Design | Asynchronous systems | CI/CD | Cassandra | Data ModelingSenior-level Full TimeBangalore,India1d ago
-
Senior Software Engineering Lead - Java, Python, AI/ML INR 3000K-5000KAgentic AI | Application Programming | Application Programming Interfaces | Automation | AzureCareer development opportunities | Comprehensive benefitsSenior-level Full TimePrimary location: Gurgaon, India1d ago
-
Data Architect Advisory, AVP INR 1500K-2500KAPI Integration | Access Control | Azure Data | Azure Data Lake | BigQueryAccident insurance | Best in class leave policy | Certification sponsorship | Childcare assistance reimbursement | Employee assistance programSenior-level Full TimePune - Margarpatta, India1d ago
-
Associate Architect - Data INR 1500K-2000KAmazon Web Services | Apache Spark | Azure Data | Azure Data Factory | Azure DevOpsMid-level Full TimeIN KA Bengaluru, India R1d ago
-
Assoc. Dir ,SAP Data and Analytics Tech Architecture INR 2771K-4300KAPIs | Analytics Cloud | Batch Data Processing | Batch data | Business Data CloudSenior-level Full TimeIND - Telangana - Hyderabad (Hitec … R1d ago
-
AWS | AWS Glue | Amazon Redshift | Amazon S3 | Cluster configurationSenior-level Full TimeIN: Lilly Bengaluru, India1d ago
-
Senior-level Full TimeHyderabad, IN, 5000813d ago
-
Generative AI Architect- Manager INR 2000K-5000KAI Search | Agent Orchestration | Agent communication | Agent communication protocols | Agentic SystemsSenior-level Full TimeBangalore (SDC) - Bagmane Tech Park, …3d ago
-
DATA Architect - Lead Systems Engineer INR 2771K-4132KAgile | Apache Airflow | Apache Kafka | Apache Spark | Audit LoggingSenior-level Full TimeIndia3d ago
-
AI Foundry | AWS | AWS SageMaker | Agentic AI | Amazon Bedrock401k match | Career development resources | Dental insurance | Flexible work arrangements | Health Savings Account medical planSenior-level Full TimeIndia3d ago
-
Senior-level Full TimePune, Maharashtra, India3d ago
-
AI Operations | AI workflows | AWS | C++ | Cloud platformSenior-level Full TimeNew Delhi, Delhi, India4d ago
-
AWS S3 | AWS S3 API | Apache Flink | Apache Hudi | Apache IcebergCross-functional collaboration | High visibility thought leadership | Industry conference participation | Influence product strategy | Remote workExecutive-level Full TimeIndia R4d ago
-
Azure Data | Azure Data Factory | Azure Synapse | Azure Synapse Analytics | Data ArchitectureSenior-level Full TimeChennai, Tamil Nadu4d ago
-
Azure | Azure Data | Azure Data Factory | Azure Synapse | Azure Synapse AnalyticsSenior-level Full TimeBengaluru, karnataka4d ago
-
Solution Architect- Data and AI INR 2000K-4300KAzure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake Storage | Azure Data Lake Storage Gen2Senior-level Full TimePune, India4d ago
-
Data & AI Solution Architect USD 9KDBT | Data product | Data product design | Databricks | Dimensional ModelingRemote workSenior-level Full TimeIndia - Bengaluru R4d ago
-
Manager, AI & Data Engineering INR 1860K-3465KAWS | Amazon Web Services | Batch Processing | CI/CD | Cloud ArchitectureHealth programs | Security programs | Well-being programsSenior-level Full TimeEcospace Campus 3A, 4th Floor, Outer …4d ago
-
Senior Data Scientist and Solution Architect INR 3000K-4500KAgent Orchestration | Agentic Workflows | Bias Mitigation | Deep learning | EmbeddingsSenior-level Full TimeBangalore, India4d ago