Data Engineer - Fabric Pipelines / Orchestration
Tasks
- Build Fabric pipelines
- Build reusable patterns and automation frameworks
- Collaborate on discovery sessions to derive requirements
- Design develop and maintain data pipelines on Apache Spark
- Ensure scheduling only ETL logic in code
- Improve data quality
- Investigate and remediate pipeline failures
- Manage code with GitHub version control
- Migrate Spark job definitions from Synapse to Fabric
- Monitor and support pipelines after go live
- Optimize for performance and scalability
- Optimize pipeline performance
- Refactor and clean up existing code
- Set up data orchestration
- Work within metadata driven architecture
Perks/Benefits
Skills/Tech-stack
Apache Spark | Azure Synapse | Data Quality | Data Warehousing | Data integration | Data orchestration | Data pipeline | ETL | GitHub | Metadata Driven | Metadata-driven architecture | Microsoft Fabric | Performance Tuning | SQL | Scala | Scalability
Roles
Related jobs
-
Senior-level Full TimePune, India14h ago
-
AWS | Amazon Kinesis | Amazon Redshift | Apache Airflow | Apache BeamFree games | Hybrid work | Learning and growth opportunities | Medical insurance | Mental health supportEntry-level Full Time地点:Hyderabad, Telangana, India17h ago
-
Mid-level Full TimeLocations: Hyderabad, Telangana, India18h ago
-
AWS CloudFormation | Amazon Kinesis | Amazon Redshift | Amazon Web Services | Apache AirflowCareer growth opportunities | Complimentary games | Family leave | Healthcare coverage | Mental wellbeing supportEntry-level Full TimeLocations: Hyderabad, Telangana, India18h ago
-
Associate Distinguished Engineer (Data Governance Framework, Data Catalog Implementation) INR 1300K-2000KBusiness glossary | Compliance | Data Architecture | Data Cataloging | Data ClassificationAwareness sessions | Continuous learning opportunities | Training programsMid-level Full TimeRemote, India R19h ago
-
GCP, Bigquery, Airflow/Consultant Specialist INR 1500K-2500KAgile | Airflow | Apache Kafka | Apache NiFi | BigQuerySenior-level Full TimeHyderabad, Telangana, India R20h ago
-
GCP, Airflow/CONSULTANT SPECIALIST INR 1500K-2500KAccess Control | Apache Airflow | Apache NiFi | Audit Logging | BigQueryFlexible working | Opportunities for professional developmentSenior-level Full TimePune, Maharashtra, India R20h ago
-
Senior Manager of Software Engineering Analytics Platform INR 3000K-5000KAgile | Artificial Intelligence | CI/CD | Cloud Native | DatabricksSenior-level Full TimeHyderabad, Telangana, India20h ago
-
Senior-level Full TimeHyderabad, TS, India21h ago
-
Senior-level Full TimeHyderabad, Telangana, India21h ago
-
Mid-level Full TimeGurgaon, Haryana, India22h ago
-
Software Engineer II - UI + Python Developer + AI / ML INR 2500K-4000KAWS | Agile | CI/CD | CSS | HTMLMid-level Full TimeMumbai, Maharashtra, India23h ago
-
Data Engineer Advisor INR 3200K-4200KAlerting | Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake StorageSenior-level Full TimeHyderabad, TG, IN23h ago
-
Machine Learning Engineer - LLM & GenAI INR 2000K-5000KAPI Integration | Caching | Chroma | CrateDB | Embedding OptimizationRemote workMid-level Full TimePune, MH, India1d ago
-
Data and AI/ML Design engineer at NE-VM INR 2000K-3300KAgile | Apache Kafka | C# | CSS | Computer VisionSenior-level Full Timebangalore, India1d ago
-
Senior Data Engineer (Python, PySpark, Kafka) INR 1500K-2700KApache Kafka | Data Warehousing | Dimensional Modeling | Git | GitHubSenior-level Full TimeBangalore, India1d ago
-
Data Engineer_Sr_Python Specialist INR 1500K-2500KAPIs | Alerting | Data Augmentation | Data Lineage | Data ModelingSenior-level Full Timebangalore, India1d ago
-
Senior-level Full TimeMumbai, Maharashtra, India1d ago
-
Applied Computer Vision Engineer - Data Driven Development INR 4000K-5500KAWS | Apache Spark | Azure | Convolutional Neural Networks | Deep learningSenior-level Full Timebengaluru, India1d ago
-
AI Data Engineering and Sr Cloud Specialist INR 1500K-2500KApache Airflow | Apache Kafka | Apache Spark | Azure | CI/CDSenior-level Full Timebengaluru, India1d ago
-
AI ML Data Engineer for ADAS Sr Backend Developer INR 2000K-4000KAWS | Apache Kafka | CI/CD | Cloud platform | DockerSenior-level Full Timebangalore, India1d ago
-
Senior Machine learning Engineer INR 4000K-5500KAzure | Azure ML | Azure Storage | CI/CD | Data loadingSenior-level Full Timebangalore, India1d ago
-
Data Engineering and Cloud INR 1500K-2500KApache Airflow | Apache Kafka | Apache Spark | Azure | CCPASenior-level Full Timebengaluru , India1d ago
-
Senior-level Full TimeIndia, Telangana, Hyderabad1d ago
-
Principal Consultant - Data & AI INR 3000K-4800KAgile | Apache Spark | Azure | Azure Data | Azure Data FactorySenior-level Full TimeIndia, Telangana, Hyderabad; India, Karnataka, Bangalore; …1d ago