Senior Manager
INR 1500K-3000K (estimate) Senior-level Full Time
Tasks
- Automate Databricks infrastructure deployment Databricks Asset Bundles Terraform infrastructure as code
- Build data ingestion frameworks Auto Loader COPY INTO dbt Fivetran Airbyte
- Build lakehouse architectures Delta Lake cloud storage integration Azure Data Lake Storage Kafka Event Hubs Kinesis
- Build streaming ingestion pipelines Kafka Event Hubs into Delta tables
- Configure Unity Catalog governance access control permissions metadata lineage
- Create schedule manage Databricks Jobs Workflows orchestration
- Design Databricks workspaces clusters compute pools
- Design implement Delta Lake tables partitioning Z ordering optimize vacuum file compaction
- Develop Delta Live Tables pipelines data quality expectations ETL ELT
- Develop batch and real time pipelines PySpark Spark SQL Structured Streaming
- Develop feature engineering pipelines Databricks Feature Store
- Develop reusable transformation frameworks utility libraries pipeline templates
- Enable generative AI Retrieval Augmented Generation vector search LLM fine tuning Mosaic AI
- Enable machine learning training inference GPU compute
- Enforce data quality Delta Live Tables expectations Great Expectations
- Implement CI CD pipelines Azure DevOps GitHub Actions GitLab CI
- Implement MLOps model versioning deployment A B testing Databricks Model Serving
- Implement error handling retry logging monitoring dead letter queue patterns
- Implement medallion architecture bronze/silver/gold
- Implement row level security column level masking dynamic views governance policies Unity Catalog
- Implement workspace best practices folder access controls secret management
- Integrate Databricks with Azure Data Lake Storage and cloud services
- Maintain enterprise data cataloging metadata management end to end data lineage
- Manage MLflow experiments model registry model lifecycle
- Manage schema evolution table versioning time travel change data feed
- Monitor pipeline execution cluster utilization system performance cloud costs
- Optimize Databricks cluster configurations autoscaling spot instances compute pools
- Optimize PySpark applications broadcast joins Adaptive Query Execution dynamic partition pruning caching Photon Engine
- Perform query optimization execution plan analysis caching performance tuning
- Prepare technical documentation architecture diagrams runbooks standard operating procedures
Perks/Benefits
- N/A
Skills/Tech-stack
Adaptive Query | Adaptive query execution | Airbyte | Amazon Kinesis | Asset bundles | Auto Loader | Azure Data | Azure Data Lake | Azure Data Lake Storage | Azure DevOps | Azure Event | Azure Event Hubs | CI/CD | COPY INTO | Catalog Governance | Change Data Feed | DBT | Data Lake Storage | Data feed | Databricks | Databricks (asset) | Databricks Asset Bundles | Databricks Feature Store | Databricks Model Serving | Databricks Repos | Databricks SQL | Databricks SQL Warehouses | Delta Lake | Delta Live Tables | Delta Live) | Event Hubs | Feature Store | Fine Tuning | Fivetran | GitHub Actions | GitLab CI | Great Expectations | Infrastructure Deployment | Infrastructure as Code | Kafka | LLM Fine-tuning | MLOps | MLflow | Medallion Architecture | Model Serving | Mosaic AI | Photon | Photon Engine | PySpark | Query Execution | Retrieval-Augmented Generation | SQL Warehouses | Spark | Spark SQL | Structured Streaming | Terraform | Time Travel | Unity Catalog | Unity Catalog Governance | Vector Search | “as-code”
Education
Bachelor of Engineering | Bachelor of Science | Master of Science
Related jobs
-
AWS Lambda | Amazon DynamoDB | Amazon RDS | Amazon S3 | CI/CDHybrid workSenior-level ContractHyderabad, India R2h ago
-
AI/ML Engineer INR 1500K-2500KAPIs | Azure | Continuous integration | Data Pipelines | Distributed SystemsCareer advancement programs | Certification programs | Dental & vision coverage | Education reimbursement | Employee assistance programMid-level Full TimeBangalore, Karnataka, India R7h ago
-
Mid-level Full TimeGurugram, Haryana, India R9h ago
-
Mid-level Full TimePune, Maharashtra, India R9h ago
-
AI Data Engineer III INR 3000K-4000KAWS | Agile methodologies | Amazon Redshift | Apache Airflow | CI/CDEducation assistance program | Fitness benefits | Flexible working hours | Health checkups | Hybrid work environmentSenior-level Full TimeBengaluru, Karnataka, India R10h ago
-
API Development | AWS | Agents | Angular | AzureHome office setup budget | International multicultural environment | Learning and development budget | Visa and relocation support | Work from homeSenior-level Full TimeIndia R11h ago
-
Mid-level Full TimeIndia (Remote) R11h ago
-
Lead Data Engineer INR 2500K-4500KAWS | AWS Glue | Access Control | Airflow | AzureCareer mobility | Cross portfolio opportunities | Leadership development | Learning opportunitiesSenior-level Full TimeIndia R12h ago
-
Data Science AI Engineer, Advisor INR 2000K-5000KAPI Development | AWS | Angular | Azure | Cloud platformRemote workSenior-level Full TimeBangalore, KA, IN, 560100 R16h ago
-
Azure Senior AI Engineer INR 2400K-4200KAI Search | API Development | Agentic AI | Azure AI | Azure AI SearchSenior-level Full TimeBangalore, KA, IN R16h ago
-
Mid-level Full TimeHyderabad, India R16h ago
-
Senior-level Full TimeHyderabad, India R16h ago
-
AI/ML Engineering Manager INR 1900K-2800KAWS CDK | AWS Lambda | AWS SageMaker | Amazon ECS | Amazon EKSAccess to Headspace app | Flexible vacation | Hybrid work model | Mental health days | Mental physical and financial wellbeing resourcesMid-level Full TimeIndia, Bengaluru, Karnataka R2d ago
-
Senior AI Engineer (m/f/d) INR 3242K-5000KAPIs | AWS | Agents | Angular | AzureHome office setup budget | Learning and development budget | Relocation support | Visa support | Work from homeSenior-level Full TimeRemote EMEA, Remote India R2d ago
-
BigQuery | CI/CD | Cloud Functions | Cloud Run | Cloud StorageContinuous professional development | Flexible workingMid-level Full TimeHyderabad, Telangana, India R3d ago
-
Analytics Engineer INR 2800K-4200KCI/CD | DBT | Dashboards | Data Quality | DatabricksHealth insurance | Hybrid work model | Mental health support | Paid time off | Transport allowanceSenior-level Full TimeBangalore (Hybrid) R3d ago
-
API | Apache Hadoop | Apache Kafka | Apache Spark | Azure DataHybrid workSenior-level Full TimeBangalore, Karnataka, India R3d ago
-
API | Agile | Apache Hadoop | Apache Kafka | Apache SparkHybrid work | Technical research opportunities | Training supportSenior-level Full TimeBangalore, Karnataka, India R3d ago
-
Apache Airflow | Apache Hadoop | Apache Kafka | Apache Spark | Azure DataHybrid work | Training opportunitiesSenior-level Full TimeBangalore, Karnataka, India R3d ago
-
Senior Software Engineer II (Java Microservices) INR 3000K-4500KAWS | Agile | Apache Kafka | Apache Spark | AzureCareer development | Distributed collaboration | Learning opportunities | Work from Anywhere cultureSenior-level Full TimeIndia R3d ago
-
Senior-level Full TimePune, MH, IN R3d ago
-
Data Science & Engineering Lead INR 3000K-4500KAPI Gateway | AWS Glue | AWS Lambda | Airbyte | Amazon EC2Senior-level Full TimeIndia - Remote R3d ago
-
AWS | AWS Glue | Agile | Amazon S3 | CI/CDSenior-level Full TimeHyderabad, Telangana, India - Remote R3d ago
-
Senior AI and Machine Learning Engineer INR 3000K-4680KConvolutional Neural Networks | Data Mining | Data Preprocessing | Data Visualization | Decision TreesCareer Development Programs | Flexible work arrangements | Health and wellbeing benefits | Hybrid work schedule | Inclusion initiativesSenior-level Full TimeBengaluru, Karnātaka, India R3d ago
-
A/B | A/B Testing | AWS | Apache Spark | AzureFlexibility programmes | Hybrid work | Inclusive benefits | MentorshipSenior-level Full TimeBengaluru Millenia, India R3d ago