Data Engineer
Tasks
- Build and maintain analytical warehouse data pipelines
- Create incremental models and snapshot tables
- Develop SQL transformation models
- Develop semantic layer business metrics
- Implement data quality checks with Great Expectations or Soda
- Implement governed data sharing for clients
- Ingest Kafka data into Microsoft Fabric OneLake
- Manage OneLake storage partitions and retention
- Migrate batch extracts to Kafka sourced Parquet
- Optimize Delta table layouts and query performance
- Set up CI/CD pipelines with GitHub Actions
Perks/Benefits
- N/A
Skills/Tech-stack
Apache Kafka | Avro | Avro to Delta | Azure Data | Azure Data Factory | Azure Synapse | Azure Synapse Analytics | CI/CD | Data Catalog | Data Factory | Data Lineage | Data Quality | Data Sharing | Data factory pipelines | Dataflows | Delta Lake | Delta Sharing | Fabric SQL | Fabric notebooks | File Compaction | GitHub Actions | GitOps | Great Expectations | Kafka Connect | Microsoft Fabric | OneLake | Parquet | Partitioning | PySpark | SQL | SQL Models | Semantic layer | Soda | Spark SQL | Synapse Analytics | Z-ordering
Education
Roles
Related jobs
-
DE&A - AWS - Data Engineer + Data Platform Support INR 3000K-4000KAmazon Bedrock | Amazon EC2 | Amazon EMR | Amazon S3 | Amazon SageMakerSenior-level Full TimePune, Maharashtra, India2h ago
-
Senior-level Full TimeHyderabad, India6h ago
-
Sr. Data Engineer INR 2800K-4000KCloud warehousing | DAX | Dashboard Design | Data Quality | Data StorytellingCommunity outreach programs | Employee Resource Group | Employee stock purchase program | Health and wellness incentivesSenior-level Full TimeBangalore, KA, IN, 5600377h ago
-
Senior-level Full TimeGurgaon, Haryana, India7h ago
-
Analytics Solutions Associate - Data Strategy INR 800K-1250KAWS Databricks | AWS Redshift | Data Governance | Data Privacy | Data QualityMid-level Full TimeBengaluru, Karnataka, India8h ago
-
GIS Data Engineer II INR 2000K-2400KAWS | Amazon Athena | Amazon EC2 | Amazon RDS | Amazon S3MentorshipSenior-level Full TimeMumbai, MH, India9h ago
-
Sr Engineer-Data Engg INR 1500K-2300KAzure AI | Azure SQL | Azure Synapse | Azure Synapse Analytics | DAXSenior-level Full TimeINDIA - BENGALURU - HP, IN13h ago
-
Software Engineer - Embedded & Functional Safety Software INR 1753K-3440KAgile | Automated testing | C# | C++ | CAN busSenior-level Full TimeBangalore, IN13h ago
-
Software Engineer - Robot Navigation, Localization & SLAM INR 1753K-2372K3D camera | Agile | Automated testing | C++ | CI/CDMid-level Full TimeBangalore, IN13h ago
-
Software Engineer - Robot Connectivity & Wi Fi Systems INR 1000K-2000KAgile | Automated testing | CI/CD | GRPC | GitCustomer-facing workSenior-level Full TimeBangalore, IN13h ago
-
AI Software Engineer INR 1753K-2600KAWS | Agent/tool use | Apache Spark | Azure | ContainersContinuing education program | Continuous learning | Family-friendly perks | Financial wellness programs | Flexible time offEntry-level Full TimeIN - HYDERABAD ORION, India13h ago
-
Apache Spark | Azure Blob | Azure Blob Storage | Azure Data | Azure Data FactoryFlexibility programmes | Inclusive benefits | Mentorship | Wellbeing supportSenior-level Full TimeNoida, India13h ago
-
IN_Manager_SSIS_Data & Analytics_Advisory_Noida INR 1500K-2000KAWS | Azure | CI/CD | Cloud Data | Cloud Data PlatformsFlexibility programmes | Inclusive benefits | MentorshipMid-level Full TimeNoida, India13h ago
-
EY - GDS Consulting - AIA - Gen AI - Staff INR 3334K-5000KAWS | AWS Bedrock | Azure OpenAI | CD | CISenior-level Full TimePune, MH, IN, 41101413h ago
-
EY - GDS Consulting - AI And DATA -Snowflake Architect -Manager INR 3000K-4000KAgile | Batch Processing | Cloud Data | Cloud Data Warehouse | DBTCareer development | Coaching and feedback | Flexible work arrangementSenior-level Full TimeHyderabad, TG, IN, 50008113h ago
-
AI Software Engineer INR 1753K-2600KAWS | Agentic AI | Azure | CI/CD | Code reviewContinuing education program | Continuous learning | Family-friendly perks | Financial wellness programs | Flexible time offEntry-level Full TimeIN - HYDERABAD ORION, India13h ago
-
Software Engineer INR 1400K-2000KAgile | Cloud services | Code review | Data Modeling | Database DesignMid-level Full TimePune, India21h ago
-
Mid-level Full TimeGurugram, Haryana, India22h ago
-
Amazon Web Services | Apache Airflow | Apache Kafka | Apache Spark | Cloud platformContract employment | Hybrid work | VIRTUAL interviewSenior-level ContractBangalore North, India23h ago
-
Senior-level Full TimePune, Maharashtra, India1d ago
-
Senior-level Full TimeGurugram, Haryana, India1d ago
-
Senior-level Full TimeGurugram, Haryana, India1d ago
-
Senior Software Engineer INR 2040K-3380KAmazon Web Services | Apache Kafka | Apache Spark | Azure | BFSSenior-level Full TimeBengaluru, KA, IN1d ago
-
Senior-level Full TimeBengaluru, Karnataka, India1d ago
-
Senior Lead Software Engineer - Java, PySpark, and AWS INR 1800K-2400KAWS | AWS Glue | AWS Lake Formation | AWS Lambda | AWS Step FunctionsSenior-level Full TimeHyderabad, Telangana, India1d ago