Senior Data Engineer

Orion Innovation is a premier, award-winning, global business and technology services firm.  Orion delivers game-changing business transformation and product development rooted in digital strategy, experience design, and engineering, with a unique combination of agility, scale, and maturity.  We work with a wide range of clients across many industries including financial services, professional services, telecommunications and media, consumer products, automotive, industrial automation, professional sports and entertainment, life sciences, ecommerce, and education.

Responsibilities:

•  Build and maintain the infrastructure for data generation, collection, storage, and processing.
• Design, build, and maintain scalable data pipelines to support data flows from various sources to data warehouses and analytics platforms.
• Develop and manage ETL (Extract, Transform, Load) processes to ensure data is accurately transformed and loaded into the target systems.
• Design and optimize databases, ensuring performance, security, and scalability of data storage solutions.
• Integrate data from various internal and external sources into unified data systems for analysis.
• Work with big data technologies (e.g., Hadoop, Spark) to process and manage large volumes of structured and unstructured data.
• Implement and manage cloud-based data solutions using Azure & Fabric platforms.
• Ensure data quality by developing validation processes and monitoring for anomalies and inconsistencies.
• Work closely with data scientists, analysts, and other stakeholders to meet their data needs and ensure smooth data operations.
• Automate repetitive data processes and workflows to improve efficiency and reduce manual effort.
• Implement and enforce data security protocols, ensuring compliance with industry standards and regulations.
• Optimize data queries and system performance to handle large data sets efficiently.
• Create and maintain clear documentation of data pipelines, infrastructure, and processes for transparency and training.
• Set up monitoring tools to ensure data systems are functioning smoothly and troubleshoot any issues that arise.
• Stay updated with emerging trends and tools in data engineering and continuously improve data infrastructure.

Qualifications:

• Azure Solution Architect certification preferred
• Microsoft Fabric Analytics Engineer Associate certification preferred
• 5+ years of architecture experience in the technology Operations/Development using Azure technologies.
• Strong experience in Python and PySpark required
• Strong understanding and experience in building lake houses, data lakes and data warehouses
• Strong experience in Microsoft Fabric technologies.
• Good understanding of the Scrum Agile methodology
• Strong experience with Azure Cloud technologies
• Solid knowledge of SQL, and non-relational (NoSQL) databases
• Solid knowledge of networking, firewalls, load balancers etc.
• Exceptional communication skills and the ability to communicate appropriately with technical teams
• Familiarity with at least one of the following code build/deploy tools Azure DevOps or GitHub Actions

Orion is an equal opportunity employer, and all qualified applicants will receive consideration for employment without regard to race, color, creed, religion, sex, sexual orientation, gender identity or expression, pregnancy, age, national origin, citizenship status, disability status, genetic information, protected veteran status, or any other characteristic protected by law.

Candidate Privacy Policy

Orion Systems Integrators, LLC and its subsidiaries and its affiliates (collectively, “Orion,” “we” or “us”) are committed to protecting your privacy. This Candidate Privacy Policy (orioninc.com) (“Notice”) explains:

• What information we collect during our application and recruitment process and why we collect it;
• How we handle that information; and
• How to access and update that information.

Your use of Orion services is governed by any applicable terms in this notice and our general Privacy Policy.