Lead Data Engineer

Job Description

Lead Data Engineer

• Seeking candidates with total experience of 9+ years and 6 years into data Engineering field.
• Be an advocate with obsession for excellence delivering measurable success for Advance Auto Parts stakeholders with secure, scalable, highly available cloud architecture that leverages AWS Cloud services.
• Provides technical leadership in AWS cloud tech stack, Snowflake and any ETL tool like Talend.
• Extensive hands-on expertise in Pyspark, Airflow and RDS(Postgress).
• Capable of conceptualizing, prototyping, and articulately presenting appropriate solutions to leadership for effective decision-making.
• Ability to assess the current environment, requirements and propose Data Engineering Solutions.
• Ability to build frameworks and pipelines using AWS and PySpark.
• Skilled in crafting comprehensive solutions across AWS and Azure platforms, with demonstrated hands-on experience in design and implementation.
• Proficiency in delivering high-quality data engineering services is a must have for this role.
• Lead the entire lifecycle of project delivery, from development to production, and provide post-production warranty support.
• Able to work with platform team in problem solving and troubleshooting activities.
• Propose Standard and Best Practices by leveraging AWS Services.
• Apply creative thinking/approach to arrive at technical solutions that enhance business goals and align with corporate technology strategies.
• Help Application teams in onboarding the feeds into the built-in frameworks.
• Make sure to follow the best design practices during the implementation of ETL Frameworks.
• Demonstrate expertise in developing pipelines within Databricks and Airflow frameworks, ensuring meticulous attention to detail in establishing efficient monitoring, logging, and auditing capabilities.
• Responsible for conducting regular code reviews and performance evaluations to ensure high standards of quality and efficiency.
• Capable of cultivating effective collaboration with technical teams, delivery managers, as well as data engineers, scientists, and analysts.

Qualification:

• Bachelor’s or master’s degree in computer science or equivalent experience
• At least 9 years of experience in application design, development, and analysis
• Experience in AWS Cloud Solutions. Retail Industry experience preferred.
• Implement data transformations and data structures for data warehouse and lake/repository.
• Experience in creating Various ETL Frameworks in processing or extracting the data from cloud databases by using AWS Services by leveraging Lambda, Glue, PySpark, Step Functions, SNS, SQS, Batch and Databricks .
• Having certifications in AWS, Azure, Snowflake, and Databricks is an added advantage for this position.
• Proficiency in Git, Azure DevOps, and DevOps pipelines is a valuable asset for this position.
• Proficient in Terraform for infrastructure such as code, Python scripting for automation, shell scripting for system tasks, and containerization technologies. Additionally, experienced in orchestrating web services.
• Proven ability to be a strategic thinker to drive the necessary ownership and data governance within the organization.
• Ability to evaluate risks and provide business recommendations / solutions in a timely manner
• Should have sound knowledge in various AWS Services.
• Able to scan ETL Frameworks and propose optimizations and cost savings.
• In addition to technical proficiency, the candidate should possess the ability to mentor junior team members, providing guidance and support to foster their professional growth and development within the team.
• Understanding of and experience working in an Agile environment.
• Able to work with other cross functional team members, both onshore and offshore for solutions and development tasks
• Independent. Strong critical thinking, decision making, troubleshooting and problem-solving skills.
• Go-Getter. Possesses strong, planning, execution and multitasking skills and demonstrated ability to reprioritize accordingly. Must be able to manage quickly changing priorities while meeting deadlines.

California Residents click below for Privacy Notice: