Data Engineer

As a Data Engineer, you are responsible for the development, performance, quality, and scaling of our data pipelines, with a special focus on distributed processing and data quality.

You will be working with stakeholders throughout the company to ensure we have high quality data to power our Data Science initiatives.

You will be collaborating closely with Data Scientists to ingest and efficiently process required data and facilitate the deployment of data science models into production.

Your Responsibilities:

• Build solutions and tools to enable, organize and process large amounts of data.
• Design and implement data ingestion and transformation in cloud-based environment and in on-premise platform.
• Create and maintain optimal data pipeline architecture for both batch and real-time processing.
• Developing scalable, flexible, and high-performance distributed data pipelines with real time stream processing.
• Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability.
• Work closely with data scientists to define the best way to ingest and aggregate required data and process it efficiently, and how to get their data science models to run in production.
• Collaborate with business stakeholders to ingest new sources, cleaning and preparing the data and writing the code that connect it to the data lake or to specific application(s).
• Evaluate new technologies and build prototypes for continuous improvements in Data Engineering

Requirements

Your Qualifications:

• MSc degree (or equivalent) in Computer Science, Informatics, Information Systems or another quantitative field
• 4+ years’ experience in data engineering / architecting in big data environment both on-premise and in the cloud, with hands-on work covering the followings:

Your Proficiencies:

• Building and optimizing complex data pipelines and architectures.
• Experience architecting, designing, and implementing cloud solutions on AWS environment is a must.
• Orchestration and workflow management tools: Airflow, AWS Step Function, MLflow, Kubeflow, etc.
• Developing and managing stream-processing systems: Hadoop/Spark-Streaming, Kafka/Flink, AWS Kinesis…
• Object-oriented/object function scripting languages: Python, Java, Scala, etc.
• Deployment and provisioning automation tools e.g., Docker, Kubernetes, Terraform
• Knowledge of search engines such as Elastic search
• Technical aspects of Data Governance including glossary, metadata, data quality, master data management & data lifecycle management
• Security, authentication and authorization
• DevSecOps / MLOps and CI / CD frameworks