EY - GDS Consulting - AI and DATA - Pyspark

At EY, you’ll have the chance to build a career as unique as you are, with the global scale, support, inclusive culture and technology to become the best version of you. And we’re counting on your unique voice and perspective to help EY become even better, too. Join us and build an exceptional experience for yourself, and a better working world for all. 

EY GDS – Data and Analytics (D&A) – PYSPARK - GIG

As part of our EY-GDS D&A (Data and Analytics) team, we help our clients solve complex business challenges with the help of data and technology. We dive deep into data to extract the greatest value and discover opportunities in key business and functions like Banking, Insurance, Manufacturing, Healthcare, Retail, Manufacturing and Auto, Supply Chain, and Finance.

The opportunity

We are currently seeking a seasoned Azure Data Engineer with proven experience in Databricks PySpark to join our team of professionals. The successful candidate will play a key role in our Metadata Management team, providing cutting-edge data management strategies and leveraging their strong analytical skills to solve complex problems.

Primary Skills / Must have:

• Should have strong programming skills in Python.
• Experience in creating large scale data processing pipelines using a Python and Spark based framework.
• Work with different aspects of the Spark ecosystem, including Spark SQL, DataFrames, Datasets, and Streaming
• Should possess strong SQL skills.
• Excellent understanding of Unix ecosystem and should have experience in creating the shell scripts.
• Excellent understanding of Hive/Hadoop ecosystem.
• Solid Understanding of data engineering concepts and best practices.
• Excellent understanding of Job Scheduling mechanisms like Autosys, TWS.
• Excellent problem solving and analytical skills.
• Excellent verbal and written communication skills.
• Experience in optimizing large data loads.

Secondary Skills / Desired skills

• Exposure to an Agile Development environment would be a plus.
• Strong understanding of Data warehousing domain.
• Ability to architect an ETL solution and data conversion strategy.
• Good understanding of dimensional modelling.

Roles and Responsibilities

As an ETL/ Python developer, the candidate is expected to

• Design and development of highly optimized and scalable ETL applications using Python and Spark.
• Undertaking end-to-end project delivery (from inception to post-implementation support), including review and finalization of business requirements, creation of functional specifications and/or system designs, and ensuring that end-solution meets business needs and expectations.
• Development of new transformation processes to load data from source to target, or performance tuning of existing ETL code (mappings, sessions).
• Analysis of existing designs and interfaces and applying design modifications or enhancements
• Coding and documenting data processing scripts and stored procedures.
• Providing business insights and analysis findings for ad-hoc data requests
• Testing software components and complete solutions (including debugging and troubleshooting) and preparing migration documentation.
• Providing reporting-line transparency through periodic updates on project or task status.
• Should have 5-7 years work exp.

EY | Building a better working world 

 
EY exists to build a better working world, helping to create long-term value for clients, people and society and build trust in the capital markets.  

 
Enabled by data and technology, diverse EY teams in over 150 countries provide trust through assurance and help clients grow, transform and operate.  

 
Working across assurance, consulting, law, strategy, tax and transactions, EY teams ask better questions to find new answers for the complex issues facing our world today.