Lead/Senior Data Engineer

Hyderabad, India

DATAECONOMY

Enabling Businesses to Monetize Data at Data Speeds with cutting edge Technology Services and Solutions. Big Data Management, Cloud enablement, Data Science, etc..

View all jobs at DATAECONOMY

Apply now Apply later

We are seeking a highly experienced and hands-on Lead/Senior Data Engineer to architect, develop, and optimize data solutions in a cloud-native environment. The ideal candidate will have 7–12 years of strong technical expertise in AWS Glue, PySpark, and Python, along with experience designing robust data pipelines and frameworks for large-scale enterprise systems. Prior exposure to the financial domain or regulated environments is a strong advantage.

Key Responsibilities:

  • Solution Architecture: Design scalable and secure data pipelines using AWS Glue, PySpark, and related AWS services (EMR, S3, Lambda, etc.)

  • Leadership & Mentorship: Guide junior engineers, conduct code reviews, and enforce best practices in development and deployment.

  • ETL Development: Lead the design and implementation of end-to-end ETL processes for structured and semi-structured data.

  • Framework Building: Develop and evolve data frameworks, reusable components, and automation tools to improve engineering productivity.

  • Performance Optimization: Optimize large-scale data workflows for performance, cost, and reliability.

  • Data Governance: Implement data quality, lineage, and governance strategies in compliance with enterprise standards.

  • Collaboration: Work closely with product, analytics, compliance, and DevOps teams to deliver high-quality solutions aligned with business goals.

  • CI/CD Automation: Set up and manage continuous integration and deployment pipelines using AWS CodePipeline, Jenkins, or GitLab.

  • Documentation & Presentations: Prepare technical documentation and present architectural solutions to stakeholders across levels.



Requirements

Required Qualifications:

  • 7–12 years of experience in data engineering or related fields.

  • Strong expertise in Python programming with a focus on data processing.

  • Extensive experience with AWS Glue (both Glue Jobs and Glue Studio/Notebooks).

  • Deep hands-on experience with PySpark for distributed data processing.

  • Solid AWS knowledge: EMR, S3, Lambda, IAM, Athena, CloudWatch, Redshift, etc.

  • Proven experience in architecture and managing complex ETL workflows.

  • Proficiency with Apache Airflow or similar orchestration tools.

  • Hands-on experience with CI/CD pipelines and DevOps best practices.

  • Familiarity with data quality, data lineage, and metadata management.

  • Strong experience working in agile/scrum teams.

  • Excellent communication and stakeholder engagement skills.

Preferred/Good to Have:

  • Experience in financial services, capital markets, or compliance systems.

  • Knowledge of data modeling, data lakes, and data warehouse architecture.

  • Familiarity with SQL (Athena/Presto/Redshift Spectrum).

  • Exposure to ML pipeline integration or event-driven architecture is a plus.



Benefits


  • Flexible work culture and remote options

  • Opportunity to lead cutting-edge cloud data engineering projects

  • Skill-building in large-scale, regulated environments.



Apply now Apply later

* Salary range is an estimate based on our AI, ML, Data Science Salary Index 💰

Job stats:  0  0  0

Tags: Agile Airflow Architecture Athena AWS AWS Glue CI/CD Data governance Data pipelines Data quality Data warehouse DevOps Engineering ETL GitLab Jenkins Lambda Machine Learning Pipelines PySpark Python Redshift Scrum SQL

Perks/benefits: Flex hours

Region: Asia/Pacific
Country: India

More jobs like this