Data Engineer

We are looking for a talented Data Engineer who is passionate about data in all its forms. In this role, you will be part of a talented team building modern data pipelines for ingesting, processing, and distributing large datasets using cloud-based and open-source technologies. The ideal candidate has broad experience in building data pipelines that scale across many data sources and consumers, along with presenting the data in visualization tools. You should have previously worked in agile teams building cloud-based analytics platforms and delivering solutions in an iterative, team-focused manner.

Responsibilities

• Create and maintain data pipelines that ingest, transform, and stage data for use in analytics processes.
• Build integrations with external systems using common integration patterns and tools (e.g., RESTful APIs, message queues like Kafka, ETL tools like Apache NiFi).
• Integrate with various source systems across the organization to pull data into a data lake for processing (e.g., CRM systems, third-party APIs).
• Understand the data model across multiple business domains and appropriately map it into cross-functional reports and dashboards.
• Develop and manage data processing services on Linux servers (e.g., cron jobs, shell scripts, Python scripts).

Skills and Requirements

• 7+ years of overall professional software development experience.
• 5+ years of hands-on Python development experience using pandas, SQLAlchemy, and Django.
• 5+ years of relational and columnar database development experience with strong knowledge of writing complex SQL and stored procedures.
• Professional experience in core AWS services and architecture supporting data processing systems (e.g., S3, Redshift, Lambda).
• Strong knowledge of open-source libraries for data processing like Airflow.
• Ability to clearly explain typical data processing architectures and their benefits and drawbacks.
• Familiarity with containers and microservice architectures (e.g., Docker, Kubernetes).
• Ability to understand complex business processes and data, ideally within financial companies.
• Experience with REST API development.

Preferred Qualifications

• Experience with cloud-based analytics platforms.
• Familiarity with DevOps practices and tools.
• Knowledge and automated testing and data quality tools

The estimated base salary range for this position is $175,000 to $250,000, which is specific to New York and may change in the future. Millennium pays a total compensation package which includes a base salary, discretionary performance bonus, and a comprehensive benefits package. When finalizing an offer, we take into consideration an individual’s experience level and the qualifications they bring to the role to formulate a competitive total compensation package.