Senior Manager
Bengaluru, Karnataka, India
R
INR 1500K-3000K (estimate) Senior-level Full Time
Tasks
- Automate Databricks infrastructure deployment with Terraform and DABs
- Build Delta Lake tables and optimization
- Build data ingestion frameworks with Auto Loader and COPY INTO
- Build feature engineering pipelines with Databricks Feature Store
- Build lakehouse architectures with Delta Lake
- Build streaming ingestion pipelines to Delta tables
- Configure and manage Unity Catalog
- Create and manage Databricks Jobs and Workflows
- Design and maintain Databricks workspaces and clusters
- Develop Delta Live Tables pipelines
- Develop batch and real time pipelines with PySpark and Spark SQL
- Develop reusable transformation frameworks and utilities
- Enable generative AI using RAG vector search and LLM fine tuning
- Implement CI CD pipelines for Databricks notebooks and workflows
- Implement MLOps with model serving and A B testing
- Implement error handling logging monitoring and DLQ
- Implement medallion architecture
- Implement model training inference and GPU compute enablement
- Integrate with Azure Data Lake Storage and streaming platforms
- Manage Databricks secrets and access controls
- Manage schema evolution Time Travel and Change Data Feed
- Monitor pipeline execution and cluster utilization
- Optimize Databricks cluster configurations
- Optimize PySpark applications performance
- Set up MLflow experiment tracking and model registry
Perks/Benefits
- N/A
Skills/Tech-stack
ACID | ADLS | AQE | AWS Secrets | AWS Secrets Manager | Adaptive Query | Adaptive query execution | Airbyte | Amazon Kinesis | Auto Loader | Azure Data | Azure Data Lake | Azure Data Lake Storage | Azure DevOps | Azure Event | Azure Event Hubs | Azure Key Vault | CI/CD | COPY INTO | Change Data Feed | DBT | Data Lake Storage | Data feed | Databricks | Databricks Feature Store | Databricks Model Serving | Databricks Secrets | Delta Lake | Delta Live Tables | Delta Live) | Event Hubs | Feature Store | Fivetran | GitHub Actions | GitLab CI | Great Expectations | Infrastructure as Code | Kafka | Key Vault | MLflow | Medallion Architecture | Model Serving | Photon Engine | PySpark | Query Execution | Secrets Manager | Spark SQL | Structured Streaming | Terraform | Time Travel | Unity Catalog | “as-code”
Education
Bachelor of Engineering | Bachelor of Science | Master of Science
Related jobs
-
Software Development Specialist INR 940K-940KAmazon CloudWatch | Amazon DynamoDB | Amazon S3 | Amazon Web Services | Azure DevOpsMid-level Full TimeBangalore, KA, IN R8h ago
-
Senior Manager INR 1500K-2800KAWS Glue | AWS Secrets | AWS Secrets Manager | Adaptive Query | Adaptive query executionHybrid work modeSenior-level Full TimePune, Maharashtra, India R9h ago
-
Java API & Microservices Engineer / Architect INR 2500K-3500KAPI Testing | Azure | CI/CD | Cassandra | DatadogSenior-level Full TimeBangalore, KA, IN R16h ago
-
Senior-level Full TimeBangalore, KA, IN R16h ago
-
Lead Data Engineer INR 3000K-4000KAPI | Agile | Amazon Web Services | Azure | Azure SynapseHybrid workSenior-level Full TimeIN025 Embassy Manyata Business Park, India R16h ago
-
Senior Data Platform Engineer INR 1500K-3000KARM Templates | AWS | Azure DevOps | Azure DevOps Services | Azure EntraHybrid workSenior-level Full TimeIN025 Embassy Manyata Business Park, India R16h ago
-
Senior DevOps Engineer INR 3000K-4000KAlerting | Apache Spark | BigQuery | CI/CD | DataflowFlexible working | Hybrid workingSenior-level Full TimeHyderabad Knowledge City (LTC), India R16h ago
-
Junior AI Specialist INR 300K-420KAWS | Apache Airflow | DBT | Data Modeling | Deep learningCertification programs | Health insurance | Language courses | Mentorship | Professional developmentEntry-level Full TimePune, India R1d ago
-
Senior-level Full TimeBengaluru East, Karnataka, India R1d ago
-
Staff Data Engineer INR 3242K-4200KCloud Data | Cloud data platform | Continuous integration | Data Engineering | Data QualityFlexible work hours | Mentorship | Remote work optionsSenior-level Full TimeHyderabad, Telangana , India R1d ago
-
AI Observability | Agent evaluation | Artificial Intelligence | CI/CD | Continuous EvaluationSenior-level Full TimeHyderabad, India R1d ago
-
Senior Data Engineer- Remote, India INR 1500K-2600KApache Airflow | Apache Kafka | Asset bundles | Azure | Azure DataRemote work in IndiaSenior-level Full TimeIndia R1d ago
-
Senior Data Engineer - Full Stack INR 1500K-2700KAWS | Alerting | Apache Spark | Azure | CI/CDEducation wellness and lifestyle support | Generous time-off policies | Top shelf benefitsSenior-level Full TimeIndia Remote R1d ago
-
Testing Engineering Specialist Advisor INR 2475K-3380K.NET | AI Search | AWS | Application Insights | AzureSenior-level Full TimeNoida, UP, IN R1d ago
-
Systems Integration Specialist Advisor_Storage Cloud INR 800K-1099KAWS | Access Control | Automation | Block Storage | CI/CDMid-level Full TimeBangalore, KA, IN R1d ago
-
AVP - Cloud Database Automation Engineer INR 2337K-3056KAWS RDS | Ansible | Auto Scaling | Cassandra | Cloud ConfigFlexible work hours | Work from homeExecutive-level Full TimeHyderabad IN, India R1d ago
-
AI Engineer INR 1500K-4000KAI Agents | API Design | Amazon Web Services | Async task queues | AzureRemote workMid-level Full TimeRemote - Pune, Maharashtra, India R1d ago
-
AI Engineer INR 1800K-3500KAI Foundry | AI Search | AWS Bedrock | Autogen | Azure AIHybrid work model | Onsite work at designated location | Remote work flexibilityNone Full TimeHyderabad, Telangana, India R1d ago
-
Software Engineer II, Analytics Operations INR 885K-1500KAWS Glue | Airflow | Amazon Athena | Amazon Redshift | Apache SparkMid-level Full TimeHyderabad, India R1d ago
-
Machine Learning Engineer 3 INR 1500K-3000KAttention | Automated retraining | CI/CD | Contrastive Learning | DDPMid-level Full TimeNoida, India R1d ago
-
Machine Learning Engineer 3 INR 1500K-3000KApache Spark | Attention Mechanism | Automated retraining | CI/CD | CheckpointingMid-level Full TimeBangalore, India R1d ago
-
Machine Learning Engineer INR 1860K-3000KApache Spark | Attention Mechanisms | Automated retraining | CI/CD | Contrastive LearningMid-level Full TimeBangalore, India R1d ago
-
Machine Learning Engineer 4 INR 3242K-5000KApache Spark | Attention | Automated retraining | CI/CD | CheckpointingSenior-level Full TimeNoida, India R1d ago
-
Mid-level Full TimeRemote Pune-Baroda, India R1d ago
-
Senior Data Engineer INR 3242K-4580KAgile | DBT | Data Lakes | Data Marts | Data ModelingCasual dress code | Financial health resources | Flexible work arrangements | Food benefits | Hybrid work scheduleSenior-level Full TimePune, IN, India R1d ago