Senior Data Engineer

Purpose of Role

A Senior Data Engineer specializes in preparing database models and big data infrastructure for analytical or operational uses. They are responsible for designing and creating systems that collect, manage, and convert raw data into usable information for data analysts and business analysts to interpret. Their work enables companies to make smarter decisions and optimize their operations.

A Senior Data Engineer develops and maintains scalable data pipelines and builds out new API integrations to support increasing data volume and complexity. They collaborate with data analysts and business teams to improve data models that feed business intelligence tools, increasing data accessibility and fostering data-driven decision-making.

They play a crucial role in implementing software and methodologies for data correction, reconciliation, and quality checking.

What you will do:

• Design and implement ETL processes for data transformation and preparation.
• Managing data warehousing solutions.
• Manage data pipelines for analytics and operational use.
• Ensuring data quality and consistency across various sources.
• Ensure data accuracy and integrity across multiple sources and systems.

You will fit in this job if you:

• Minimum of 4 years in data engineering within a production environment.
• Advanced knowledge of Python and Linux shell scripting.
• Database Expertise:
  • Extensive experience with SQL databases such as MySQL and PostgreSQL.
  • Proficient with NoSQL databases like BigQuery.
• Data Pipeline and ETL Development: Proven ability to design, implement, and manage ETL processes for data transformation and preparation.
• Stream Processing Systems: Strong experience building and managing stream processing systems using Kafka.
• Data Quality Management: Strong knowledge in ensuring data quality, consistency, and integrity across multiple sources and systems.
• Problem-Solving Skills: Proven analytical and problem-solving skills, particularly in identifying data inconsistencies and errors.
• Attention to Detail: High attention to detail in handling millions of daily transactions and non-transactional data.
• Collaboration: Experience working collaboratively with data analysts, business analysts, and other stakeholders to improve data models and support data-driven decision-making.

Bonus Qualifications and Skills:

• Machine Learning: Basic understanding or experience with machine learning concepts and applications.
• Generative AI: Exposure to or experience with Generative AI technologies and their applications.