Senior Data Engineer - R01563934
Bangalore, Karnataka, India
INR 1000K-2000K (estimate) Senior-level Full Time
Tasks
- Build Databricks workflows for orchestration
- Build curated datasets for analytics
- Build logging monitoring and alerting
- Design Medallion Architecture
- Design scalable ETL/ELT pipelines
- Develop batch and streaming data pipelines
- Enable ACID transactions
- Enforce schema evolution and CDC
- Implement CI/CD for data pipelines
- Implement Delta Lake performance optimization
- Implement Delta Live Tables
- Implement data quality validation
- Ingest data from databases APIs and files
- Troubleshoot pipeline failures and optimize performance
- Use Unity Catalog for governance and lineage
Perks/Benefits
- N/A
Skills/Tech-stack
ACID | Airflow | Apache Spark | Avro | Azure DevOps | Broadcast joins | CI/CD | Caching | Change Data Capture | Data Capture | Data Governance | Data Modeling | Databricks | Delta | Delta Lake | Delta Live Tables | Delta Live) | ELT | ETL | Event Hubs | Git | GitHub Actions | JSON | Kafka | MLOps | MLflow | Parquet | Partitioning | PySpark | Python | SQL | Schema evolution | Snowflake Schema | Spark SQL | Spark Streaming | Star Schema | Unity Catalog | Z-Order
Education
Roles
Related jobs
-
Fullstack Data Engineer (Remote) INR 800K-2000KAPI Keys | AWS IAM | AWS S3 | AWS VPC | Apache SparkRemote work | UK shift scheduleSenior-level Full TimeMaharashtra, Pune, India R6h ago
-
Principal Consultant - GenAI Engineer INR 3000K-4144KAWS | Agentic AI | Azure | Azure OpenAI | Cloud ComputingSenior-level Full TimeIndia-Pune7h ago
-
Principal Consultant - GenAI Engineer INR 3000K-4725KAWS | Agentic AI | Azure OpenAI | Cloud platform | Deep learningSenior-level Full TimeIndia-Kolkata7h ago
-
Senior DevOps Engineer (AWS | MLOps | Kubernetes) INR 2500K-4000KAWS IAM | Amazon EKS | Amazon S3 | Amazon Web Services | Argo CDHybrid workSenior-level Full TimeMaharashtra, Pune, India7h ago
-
GenAI Azure INR 1800K-2500K.NET | API Management | ASP.Net Core | Access Control | Active DirectoryMid-level Full TimeIN-AP-Hyderabad7h ago
-
Senior-level Full TimeHyderabad, India8h ago
-
Staff Software Engineer, Cloud SQL Postgres INR 1500K-2400KC++ | Database Administration | PostgreSQL | Query Optimization | Relational databasesSenior-level Full TimeBengaluru, Karnataka, India9h ago
-
Actuarial - Data Science INR 1500K-3000KClassification | Data Preprocessing | Data Quality | Explainable AI | Feature EngineeringMid-level Full TimeBangalore, KA, India9h ago
-
Data Engineer II - Python, AWS & Pyspark INR 1500K-2200KAWS | Data Migration | Data Modeling | Data Warehouse | Data pipelineMid-level Full TimeHyderabad, Telangana, India11h ago
-
AWS | Alerting | Audit Logging | Azure | CI/CDSenior-level Full TimePune, MH, India11h ago
-
Consultant - Gen AI Engineer INR 550K-610KAPI Integration | Generative AI | Language Processing | Machine Learning | Natural LanguageEntry-level Full TimeBangalore, Karnataka, India12h ago
-
Apache Iceberg | Apache Spark | Avro | CI/CD | Cause analysisEntry-level Full TimeHyderabad, Telangana, India12h ago
-
Engineering-L2-Bengaluru-Vice President-Software Engineering INR 1500K-6000KApache Iceberg | Apache Spark | Avro | CI/CD | DatabricksExecutive-level Full TimeBengaluru, Karnataka, India13h ago
-
Software Engineer III - Data Engineer, AWS, ETL, Python INR 2000K-3500KAWS | CI/CD | Data Governance | Data Lake | Data ProcessingSenior-level Full TimeBengaluru, Karnataka, India13h ago
-
Gen AI Lead engineer INR 3000K-4800KAgentic Frameworks | CI/CD | Data Visualization | Git | LangchainSenior-level Full TimeChennai, Tamil Nadu, India13h ago
-
Engineering-L2-Bengaluru-Associate-Software Engineering INR 1800K-3200KANSI SQL | Apache Iceberg | Apache Spark | Avro | CI/CDSenior-level Full TimeBengaluru, Karnataka, India13h ago
-
Full Stack Developer :: Acon INR 2500K-3500KAI Search | AI Studio | Application Security | Arize Phoenix | Azure AISenior-level Full TimeIndia13h ago
-
Lead Assistant Manager INR 2500K-4500KASR | AWS EC2 | Amazon ECS | Amazon SageMaker | Amazon Web ServicesSenior-level Full TimeNoida, Uttar Pradesh, India14h ago
-
Data Engineer INR 2755K-3500KAPI Development | AWS | Amazon Redshift | Azure | BashHybrid working | Inclusive culture | Learning and development programmesSenior-level Full TimeHyderabad, India14h ago
-
Director of Software Engineering - Big Data, Cloud INR 2000K-6000KAI machine learning | AWS | Agile | Azure | Big DataExecutive-level Full TimeMumbai, Maharashtra, India14h ago
-
Assistant Manager-GDC INR 1500K-2000KApache Spark | Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake StorageMid-level Full TimeBangalore, Karnataka, India15h ago
-
Data Engineer- AWS+MS Fabric+ Purview INR 2000K-3500K.NET | AWS CloudFormation | AWS Lambda | Amazon EC2 | Amazon RDSSenior-level Full TimePune, Maharashtra, India16h ago
-
Software Engineer III - Databricks, SQL , Python INR 2000K-2800KApache Spark | Cloud technologies | Data Lake | Data Warehouse | DatabricksSenior-level Full TimeBengaluru, Karnataka, India16h ago
-
Senior-level Full TimeBangalore, Karnataka, India17h ago
-
Senior-level Full TimeBangalore, India19h ago