Data Engineer – BFSI Domain
Mumbai, MH, India
Expleo
Expleo is a trusted partner for end-to-end, integrated engineering, quality services and management consulting for digital transformation.Overview
Data Engineer – BFSI Domain
Responsibilities
Key Responsibilities
- Develop and optimize ETL/ELT pipelines using Apache Spark, Apache Iceberg and Python
- Orchestrate workflows with Apache Airflow and NiFi
- Model, ingest and curate structured and semi-structured data in data lakes and warehouses
- Implement and maintain data versioning, schema evolution and partition strategies in Iceberg
- Build and maintain Google Cloud data solutions (BigQuery, Dataflow, Pub/Sub, Cloud Storage)
- Monitor job performance, troubleshoot data quality issues and tune cluster resources
- Collaborate on CI/CD for data infrastructure (Terraform, Kubernetes, Docker)
- Document pipelines, schemas and operational runbooks
Required Skills & Experience
- 5+ years in a Data Engineering role
- Hands-on with Apache Spark, Apache Iceberg, Python and SQL
- Experience authoring DAGs in Airflow and streaming/batch flows in NiFi
- Solid understanding of data modeling, partitioning and indexing strategies
- Familiarity with Linux, Git and RESTful APIs
- Strong debugging, performance-tuning and troubleshooting skills
Qualifications
Key Responsibilities
- Develop and optimize ETL/ELT pipelines using Apache Spark, Apache Iceberg and Python
- Orchestrate workflows with Apache Airflow and NiFi
- Model, ingest and curate structured and semi-structured data in data lakes and warehouses
- Implement and maintain data versioning, schema evolution and partition strategies in Iceberg
- Build and maintain Google Cloud data solutions (BigQuery, Dataflow, Pub/Sub, Cloud Storage)
- Monitor job performance, troubleshoot data quality issues and tune cluster resources
- Collaborate on CI/CD for data infrastructure (Terraform, Kubernetes, Docker)
- Document pipelines, schemas and operational runbooks
Required Skills & Experience
- 5+ years in a Data Engineering role
- Hands-on with Apache Spark, Apache Iceberg, Python and SQL
- Experience authoring DAGs in Airflow and streaming/batch flows in NiFi
- Solid understanding of data modeling, partitioning and indexing strategies
- Familiarity with Linux, Git and RESTful APIs
- Strong debugging, performance-tuning and troubleshooting skills
Essential skills
Key Responsibilities
- Develop and optimize ETL/ELT pipelines using Apache Spark, Apache Iceberg and Python
- Orchestrate workflows with Apache Airflow and NiFi
- Model, ingest and curate structured and semi-structured data in data lakes and warehouses
- Implement and maintain data versioning, schema evolution and partition strategies in Iceberg
- Build and maintain Google Cloud data solutions (BigQuery, Dataflow, Pub/Sub, Cloud Storage)
- Monitor job performance, troubleshoot data quality issues and tune cluster resources
- Collaborate on CI/CD for data infrastructure (Terraform, Kubernetes, Docker)
- Document pipelines, schemas and operational runbooks
Required Skills & Experience
- 5+ years in a Data Engineering role
- Hands-on with Apache Spark, Apache Iceberg, Python and SQL
- Experience authoring DAGs in Airflow and streaming/batch flows in NiFi
- Solid understanding of data modeling, partitioning and indexing strategies
- Familiarity with Linux, Git and RESTful APIs
- Strong debugging, performance-tuning and troubleshooting skills
Experience
-
Required Skills & Experience
- 5+ years in a Data Engineering role
- Hands-on with Apache Spark, Apache Iceberg, Python and SQL
- Experience authoring DAGs in Airflow and streaming/batch flows in NiFi
- Solid understanding of data modeling, partitioning and indexing strategies
- Familiarity with Linux, Git and RESTful APIs
- Strong debugging, performance-tuning and troubleshooting skills
* Salary range is an estimate based on our AI, ML, Data Science Salary Index 💰
Job stats:
0
0
0
Categories:
Deep Learning Jobs
Engineering Jobs
Tags: Airflow APIs BigQuery CI/CD Dataflow Data quality Docker ELT Engineering ETL GCP Git Google Cloud Kubernetes Linux NiFi Pipelines Python Spark SQL Streaming Terraform
Region:
Asia/Pacific
Country:
India
More jobs like this
Explore more career opportunities
Find even more open roles below ordered by popularity of job title or skills/products/technologies used.
Power BI Developer jobsBI Developer jobsPrincipal Data Engineer jobsSr. Data Engineer jobsStaff Data Scientist jobsPrincipal Software Engineer jobsStaff Machine Learning Engineer jobsData Science Intern jobsDevOps Engineer jobsData Science Manager jobsJunior Data Analyst jobsSoftware Engineer II jobsData Manager jobsData Analyst Intern jobsLead Data Analyst jobsStaff Software Engineer jobsAccount Executive jobsBusiness Data Analyst jobsSr. Data Scientist jobsData Specialist jobsAI/ML Engineer jobsSenior Backend Engineer jobsData Governance Analyst jobsBusiness Intelligence Analyst jobsData Engineer III jobs
Consulting jobsMLOps jobsAirflow jobsOpen Source jobsEconomics jobsLinux jobsKPIs jobsTerraform jobsJavaScript jobsKafka jobsGitHub jobsData Warehousing jobsPostgreSQL jobsRDBMS jobsNoSQL jobsComputer Vision jobsScikit-learn jobsGoogle Cloud jobsStreaming jobsClassification jobsPrompt engineering jobsBanking jobsPhysics jobsRAG jobsOracle jobs
Hadoop jobsPandas jobsdbt jobsData warehouse jobsBigQuery jobsR&D jobsScala jobsLooker jobsGPT jobsReact jobsDistributed Systems jobsPySpark jobsScrum jobsIndustrial jobsCX jobsLangChain jobsMicroservices jobsELT jobsSAS jobsJira jobsRedshift jobsOpenAI jobsJenkins jobsModel training jobsRobotics jobs