Senior Manager
Bengaluru, Karnataka, India
R
INR 1500K-3000K (estimate) Senior-level Full Time
Tasks
- Automate Databricks infrastructure deployment with infrastructure as code
- Build CI CD pipelines for notebooks jobs and workflows
- Build and maintain Medallion architecture for lakehouse data
- Build feature engineering pipelines with Databricks Feature Store
- Build ingestion frameworks using Auto Loader and COPY INTO
- Build lakehouse integrations with cloud storage and streaming systems
- Build streaming ingestion pipelines into Delta tables
- Configure and manage Unity Catalog for governance and access control
- Create schedule and manage Databricks jobs workflows and orchestration
- Design Delta Lake tables and optimize storage
- Design and maintain Databricks workspaces clusters and compute pools
- Develop Delta Live Tables pipelines with data quality expectations
- Develop batch and real time pipelines using PySpark and Spark SQL
- Develop reusable transformation frameworks and pipeline templates
- Enable ML workloads for training inference and GPU compute
- Enable generative AI solutions including RAG and vector search
- Enforce data quality with Delta Live Tables expectations and Great Expectations
- Enforce governance with row level security column masking and dynamic views
- Implement MLOps best practices including deployment and A B testing
- Implement error handling retry logging monitoring and DLQ patterns
- Implement workspace best practices including secrets management
- Maintain metadata and data lineage
- Manage MLflow tracking and model registry
- Manage schema evolution Time Travel and Change Data Feed
- Monitor pipeline execution cluster utilization and cloud costs
- Optimize Databricks cluster configurations for performance and cost
- Optimize PySpark applications for query performance
- Perform query optimization execution plan analysis and performance tuning
- Prepare technical documentation architecture diagrams and runbooks
Perks/Benefits
- N/A
Skills/Tech-stack
AWS Secrets | AWS Secrets Manager | Adaptive Query | Adaptive query execution | Apache Spark | Asset bundles | Auto Loader | Azure DevOps | Azure Key Vault | CI/CD | COPY INTO | Change Data Feed | Data feed | Databricks | Databricks (asset) | Databricks Asset Bundles | Databricks Feature Store | Databricks Jobs | Databricks Model Serving | Databricks Workflows | Dead Letter | Dead Letter Queue | Delta Lake | Delta Lake Optimization | Delta Live Tables | Delta Live) | Feature Store | Fine Tuning | GitHub Actions | GitLab CI | Infrastructure as Code | Key Vault | LLM Fine-tuning | MLOps | MLflow | MLflow Model Registry | Medallion Architecture | Model Registry | Model Serving | Mosaic AI | Photon | Photon Engine | PySpark | Query Execution | Query Language | RAG | Retrieval-Augmented Generation | SQL window functions | Schema evolution | Secrets Manager | Spark SQL | Structured Query | Structured Query Language | Structured Streaming | Terraform | Time Travel | Unity Catalog | Vector Search | Window Functions | Z-ordering | “as-code”
Education
Bachelor of Engineering | Bachelor of Science | Master of Science
Related jobs
-
Senior Data Engineering Lead INR 2000K-3900KAWS CloudFormation | AWS Glue | AWS Lambda | AWS Step Functions | Amazon AthenaFamily well-being benefits | Health benefits | Hybrid remote/in-office | Paid time off | Work-life balanceSenior-level Full TimeHyderabad, India R5h ago
-
Senior Technical Anchor INR 4000K-5500KApache Spark | Batch Processing | BigQuery | CI/CD | Cloud BuildSenior-level Full TimeChennai, Tamil Nadu, India R7h ago
-
Senior DevOps Consultant INR 3000K-4500KAKS | API Management | Application Insights | Azure | Azure ContainerCareer coaching | Employee assistance programme | Flexible working hours | Hybrid work | TrainingSenior-level Full TimeBengaluru, Karnataka, India R9h ago
-
AI Agents | AWS | Anthropic | Azure | Azure OpenAIDirect customer collaboration | Freelance contract opportunity | Fully remote | Inclusive workplace | Learning opportunitiesMid-level Full TimeIndia R10h ago
-
AI Agents | AI Evaluation | Agent systems | CI/CD | DockerFlexible project-based work | Freelance contract engagement | Fully remoteSenior-level Contract Full TimeIndia R10h ago
-
Data Engineer INR 1500K-2700KAsset bundles | Azure Data | Azure Data Lake | Azure Data Lake Storage | Azure Data Lake Storage Gen2Flexible workplace accommodations | Hybrid work modelMid-level Full TimeChennai, India R11h ago
-
Specialist Solutions Architect - AI/ML INR 2000K-4300KAI guardrails | Agent Orchestration | Agentic Systems | Amazon Web Services | Artificial IntelligenceMentorship | Technical training | Travel 30 percent maximumSenior-level Full TimeMumbai, India R12h ago
-
AWS Bedrock | Amazon SageMaker | BentoML | CI/CD | DockerCollaborative environment | Exposure to modern AI ML infrastructure | Full-time employment | Remote work from India | Technical ownershipSenior-level Full TimeIndia R14h ago
-
Sr Data Engineer - ETRM INR 2771K-4064KAgile | Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake StorageFlexible working hours | Paid parental leave | Remote working option | Training and development programmesSenior-level Full TimeBangalore RMZ-ECO WORLD, India R16h ago
-
Data Scientist Engineer Advisor INR 2400K-3380KAWS | Azure | Bias Mitigation | Data Analysis | Data QualityMid-level Full TimeBangalore, KA, IN R16h ago
-
Senior Computer Scientist 1 INR 3000K-4680KCI/CD | Cloud Platforms | Docker | Generative AI | Integration AutomationSenior-level Full TimeNoida, India R16h ago
-
Data Science Engineer INR 2775K-4500KA/B | A/B Testing | Anomaly Detection | B testing | ClusteringEntry-level Full TimeBangalore, India R16h ago
-
AI Engineer INR 3672K-4800KAPI Integration | Access Controls | Agent Orchestration | CI/CD | Data Leakage PreventionAsynchronous work | Career growth | Continuous learning | Remote workMid-level Full TimeRemote, Bangalore R18h ago
-
Senior Data Platform Engineer INR 3584K-4580KAWS | Apache Kafka | Apache Spark | Azure | Cloud platformCollaborative cross site work | Healthcare benefits | Internet or cell phone reimbursement | Learning and development stipend | Travel opportunities to headquartersSenior-level Full TimeBengaluru, India R18h ago
-
BigQuery | CI/CD | Cloud Composer | Cloud IAM | Cloud NetworkingConference reimbursement | Corporate discounts | Coworking reimbursement | Equipment and workspace reimbursement | Fully remoteSenior-level Full TimeIndia R1d ago
-
Senior AI Engineer/ Senior Associate Director, Tech SME INR 1860K-2800KAPIs | AWS | Agent Orchestration | Agent systems | AzureMid-level Full TimePune, Maharashtra, India R1d ago
-
AWS Lambda | Amazon DynamoDB | Amazon RDS | Amazon S3 | CI/CDHybrid workSenior-level ContractHyderabad, India R1d ago
-
Software Engineer I INR 1500K-2500KApache Spark | CI/CD | Data Ingestion | Data Modeling | Data QualityCareer development | Community building | Learning opportunities | Work from Anywhere cultureEntry-level Full TimeIndia R1d ago
-
Apache Airflow | Apache Beam | Apache Flink | Asynchronous Messaging | BigQueryFlexible working | Opportunities for growth | Professional developmentMid-level Full TimePune, Maharashtra, India R1d ago
-
Staff Product Analytics Engineer INR 3242K-4995KAWS | Agile | Azure | Cloud billing | Data EngineeringFlexible PTO | Hybrid work model | Learning platform | Mental health counseling | Mentorship programsSenior-level Full TimeBangalore, India R1d ago
-
AI/ML Engineering Lead/(Gen AI,Python,LLM,ETL)Associate Director INR 2000K-4000KAgent Orchestration | Agentic AI | Apache Airflow | Apache Beam | Audit LoggingFlexible working | Inclusive environment | Professional developmentMid-level Full TimePune, Maharashtra, India R1d ago
-
AI/ML Engineer INR 1500K-2500KAPIs | Azure | Continuous integration | Data Pipelines | Distributed SystemsCareer advancement programs | Certification programs | Dental & vision coverage | Education reimbursement | Employee assistance programMid-level Full TimeBangalore, Karnataka, India R1d ago
-
Senior Manager INR 1500K-3000KAdaptive Query | Adaptive query execution | Airbyte | Amazon Kinesis | Asset bundlesSenior-level Full TimePune, Maharashtra, India R1d ago
-
Mid-level Full TimeGurugram, Haryana, India R1d ago
-
Mid-level Full TimePune, Maharashtra, India R1d ago