Data Engineer - Fabric Pipelines / Orchestration
Tasks
- Build Fabric pipelines
- Build reusable patterns and automation frameworks
- Collaborate on discovery sessions to derive requirements
- Design develop and maintain data pipelines on Apache Spark
- Ensure scheduling only ETL logic in code
- Improve data quality
- Investigate and remediate pipeline failures
- Manage code with GitHub version control
- Migrate Spark job definitions from Synapse to Fabric
- Monitor and support pipelines after go live
- Optimize for performance and scalability
- Optimize pipeline performance
- Refactor and clean up existing code
- Set up data orchestration
- Work within metadata driven architecture
Perks/Benefits
Skills/Tech-stack
Apache Spark | Azure Synapse | Data Quality | Data Warehousing | Data integration | Data orchestration | Data pipeline | ETL | GitHub | Metadata Driven | Metadata-driven architecture | Microsoft Fabric | Performance Tuning | SQL | Scala | Scalability
Roles
Related jobs
-
Lead Software Engineer - Data Engineering INR 3500K-5000KAWS | Airflow | Authentication | Authorization | CI/CDSenior-level Full TimeBengaluru, Karnataka, India9h ago
-
AI/ML Engineer_MS INR 2500K-5000KAI Foundry | AI Search | AWS Bedrock | Amazon QuickSight | App ServiceSenior-level Full Timetelengana, India12h ago
-
Data Analyst Technical Lead (Alteryx) INR 3000K-4000KAPI Integration | Alteryx Designer | Alteryx Server | Data Cleansing | Data ReconciliationUK shiftSenior-level Full TimeNoida, UP, India13h ago
-
Mid-level Full TimeBangalore, Karnataka, India13h ago
-
GCP Data Engineer INR 1500K-2000KAgile | Apache Airflow | Ascend.io | CI/CD | Cloud ComposerFlexible work model | Wellbeing cultureMid-level Full TimeHyderabad, India13h ago
-
Senior Software Engineer, Machine Learning INR 2500K-3500KA/B | A/B Testing | Amazon S3 | Apache Airflow | Apache SparkAccident insurance | Commuter benefits | Dental insurance | Disability insurance | Financial wellness supportSenior-level Full TimeBengaluru, India14h ago
-
Principal Engineer, Storage Data Science and Analytics INR 3200K-4590KAIOps | Anomaly Detection | Apache Spark | C# | C++Career Development Programs | Flexible work arrangements | Health and wellbeing benefitsSenior-level Full TimeBengaluru, Karnātaka, India18h ago
-
Databricks -Project Lead-Data Engg INR 1500K-2500KAzure Data | Azure Data Factory | Azure Synapse | Azure Synapse Analytics | Data FactoryClient communication support | Flexible to work US shiftsSenior-level Full TimeINDIA - PUNE - BIRLASOFT OFFICE …18h ago
-
Senior Data Engineer (Data Analytics),Hyderabad INR 3000K-4000KAPI | AWS | Agile | Amazon DynamoDB | Amazon RedshiftEqual opportunity employer | Fast track growth opportunitiesSenior-level Full TimeHyderabad - Phoenix Equinox Tower 2, …18h ago
-
Databricks - Principal Data Engineer INR 1500K-2500KApache Spark | CI/CD | Data Modeling | Data Pipeline Performance Tuning | Data WarehousingFull-time employment | Hybrid workSenior-level Full TimeBangalore, KA, IN18h ago
-
Custom Software Engineer INR 2475K-3500KAzure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake Storage | Azure SynapseSenior-level Full TimeBengaluru, BDC11A, India18h ago
-
Senior-level Full TimeBengaluru, KA, IN, 56001618h ago
-
Senior Software Engineer (India) INR 3000K-4500KAWS | AWS CDK | Agile | Asyncio | AuthenticationRemote work | Work from homeSenior-level Full TimeIndia - Remote R18h ago
-
Business Intelligence Senior Specialist INR 1200K-2040KData Modeling | Data integration | ELT | ETL | GitHybrid workingSenior-level Full TimeBangalore, KA, IN18h ago
-
Senior-level Full TimeHyderabad, Telangana, India18h ago
-
AWS | Apache Airflow | Apache Spark | Azure | Azure DataFlexibility programmes | Inclusive benefits | MentorshipSenior-level Full TimePune, India18h ago
-
Lead Engineer Data Streaming INR 1600K-3000KAmazon Redshift | Ansible | Apache Kafka | CDC | CI/CDFlexible work environment | Internal mobility | Volunteering opportunities | Well-being programs | Work-life balanceSenior-level Full TimeKA Bangalore, India18h ago
-
Senior IT Technologist - LLM Platform Engineer INR 1753K-2772KAccess Management | Automation | Automation Scripting | CI/CD | CyberArkFlexible benefits package | Short-term incentive planSenior-level Full TimeIND-TS Hyderabad Nanakramguda, India18h ago
-
Senior Data Platforms SRE INR 1500K-2500KAWS | Azure | Capacity Planning | Cause analysis | Cluster PolicyDocumentation support | Knowledge sharing | MentoringSenior-level Full TimeBangalore, India18h ago
-
AWS | Airflow | DBT | Fine Tuning | Language ModelsBonuses | Disability insurance | Life insurance | Paid parental leave | Paid time offSenior-level Full TimeRemote, India R1d ago
-
Senior Data Engineer ( Databricks) INR 2000K-3900KAWS | Agile | Artificial Intelligence | Data Lake | Data ModelingHealth insurance | Vacation benefitsSenior-level Full TimeHyderabad, Telangana, India1d ago
-
Mid-level Full TimeHadapsar, Pune, Maharashtra, IND, 4110131d ago
-
Efs Fcrm 169474 INR 1500K-2400KApache Spark | Azure DevOps | CI/CD | Data Partitioning | Data WarehousingMid-level Full TimeHyderabad, IND-TG, India1d ago
-
Planview Administration 169576 INR 1500K-2500KAlerting | Data Governance | Data Modeling | Data Validation | Data cleaningSenior-level Full TimeHyderabad, IND-TG, India1d ago
-
Senior Database Engineer - #3190 (Remote) INR 1500K-2700KAWS IAM | AWS KMS | AWS RDS | Amazon Aurora | BashOn-call rotation | Remote workSenior-level Full TimeChennai, India R1d ago