Senior Data Engineer

Ideal candidate will have a deep understanding and knowledge of advanced data engineering, predictive modeling, and deep learning. The goal is to automate processes, reduce costs, and scale operations proactively as data volume increases.

Responsibilities:

• Lead the development and optimization of data pipelines and ETL processes using Databricks.
• Collaborate with teams who work on Integrating diverse data sources and apply business rules for predictive modeling.
• Collaborate with internal teams to generate Next Best Actions (NBA) for field representatives.
• Have a deep understanding and knowledge of AI/ML and ML Ops platforms to enhance data processing capabilities.
• Work with commercial teams to process and analyze structured data inputs.
• Ensure seamless handover and collaboration with existing teams.
• Mentor junior engineers and provide technical leadership.

Required Skills and Experience:

• 7-10 years of experience in Data Engineering and related streams.
• Proficient in SQL, Python, and Spark.
• Extensive experience with Databricks on AWS.
• Strong understanding of AI/ML concepts and ML Ops platforms, particularly the Arise platform.
• Proven ability to develop and optimize data engineering pipelines.
• Experience with predictive modeling and business rule application.
• Excellent collaboration and communication skills.
• Demonstrated ability to lead projects and mentor junior engineers.

Preferred Qualifications:

• Familiarity with commercial data processing and forecasting models.
• Background in handling large-scale data operations and integration.