Lead Data Engineer

About Acerta: 

Acerta helps manufacturers optimize the quality, safety, and reliability of the products they make. Our machine-learning solutions use industrial data to deliver valuable insights that improve the entire product lifecycle—resulting in greater efficiency and satisfied customers. 

Job Summary: 

We are seeking a talented and experienced Lead Data Engineer to join our growing team. The Lead Data Engineer will play a key role in designing, building, and maintaining our data infrastructure, ensuring scalability, reliability, and performance with a team of data engineers around them. This position offers an exciting opportunity to work with large-scale datasets, cutting-edge technologies, and a passionate team of data professionals. 

Key Responsibilities: 

• Architect and Develop Data Ingestion to Acerta Platform: Architect, develop, and maintain scalable data pipelines and ETL processes to ingest, process, and analyze large volumes of structured and unstructured data from diverse sources. 

• Collaborate with Cross-Functional Teams: Collaborate with cross-functional teams to understand data requirements, identify opportunities for data-driven insights, and develop solutions to address business challenges. 

• Optimize Data Pipelines: Optimize and tune data pipelines and database queries for performance, efficiency, and reliability. 

• Implement Data Governance: Implement data governance and security best practices to ensure data integrity, privacy, and compliance with regulatory requirements. 

• Support Machine Learning Initiatives: Work closely with data scientists to support their data needs, provide access to relevant datasets, and enable advanced analytics and machine learning initiatives. 

• Stay Abreast of Industry Trends: Stay abreast of industry trends, emerging technologies, and best practices in data engineering and analytics, and apply them to improve our data infrastructure and processes. 

Skills and Qualifications: 

• Previous experience in ETL/ELT Architecture: Expertise in designing and optimizing ETL/ELT pipelines to handle various data formats (CSV, JSON, Parquet, etc.) and integrating data from multiple sources (e.g., APIs, cloud storage, client Databricks instances).
• Strong understanding of REST API principles, experience with high-volume API requests, and ability to optimize API calls and data ingestion strategies.
• Automation and Orchestration: Proficiency in using orchestration tools like Apache Airflow, or similar tools to automate and manage data workflows.
• Proficiency using Unity Catalog to establish a consistent security and governance layer, manage access control, and ensure regulatory compliance.
• Expertise in building efficient ETL/ELT workflows to enable scalable feature engineering.
• Previous experience working closely with ML/DS teams, understanding of data modeling required for various teams and products.
• Previous experience in performance testing and optimization (data load testing/performance tuning/monitoring) for various databases, and ETL pipelines.
• Technical Mentorship: Leadership in coaching junior engineers, setting best practices for coding standards, and fostering a collaborative environment for knowledge sharing and troubleshooting.

Technical stack:

1. Programming Languages: Python, JavaScript/Node.js, Spark
2. Big Data experience: Databricks
3. Data Orchestration and Automation: K8, Apache Airflow or similar.
4. APIs and Data Ingestion: Strong understanding of REST API principles, experience with high-volume API requests, and ability to optimize API calls and data ingestion strategies.
5. Large data workflow optimization: Optimizations of
6. Bonus - experience with Vector Databases.

Why Join Acerta? 

• Competitive Salary: Competitive salary and performance-based incentives. 

• Comprehensive Benefits: Comprehensive benefits package, including health insurance, and more. 

• Flexible Work Hours: Flexible work hours and remote work options. 

• Professional Growth: Opportunities for professional growth and career advancement. 

• Collaborative Environment: Collaborative and inclusive work environment with a diverse team of professionals. 

If you are passionate about data engineering and eager to contribute to Acerta’s mission, we invite you to apply for this exciting opportunity. Apply now and be part of Acerta’s mission to shape the future of automotive intelligence! 

Acerta Analytics Solutions Inc. is an equal-opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees. 

www.acerta.ai | LinkedIn | Twitter