Data Engineer

Background Information:Innovative Defense Technologies (IDT), provider of automated software testing, data analysis, and cybersecurity solutions for complex, mission-critical systems in the US Department of Defense (DOD), is seeking a Data Engineer to be based in our Arlington, VA office.    Overview: The Data Engineer position offers the opportunity to join our growing team of data scientists and analytics experts. The hire will be responsible for designing and implementing various data pipeline architectures across several projects, as well as optimizing data flow and collection for cross functional teams. The candidate must be experienced in the state-of-the-art data management technologies to support differing data requirements and objectives. Individuals must be self-directed and comfortable supporting the data needs of multiple teams, systems, and products. All applicants must be able to obtain/maintain an active U.S. Security Clearance.  Responsibilities Include:

• Design, implement, deploy, and maintain optimal data pipeline and data management architectures.
• Assemble large, complex data sets that meet functional / non-functional project requirements
• Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL, streaming and ‘big data’ technologies
• Implement data pipelines to ingest data to the platform, standardize and transform the data
• Support the development of analytics tools that utilize the data pipeline to provide actionable insights into the data and optimization of objectives.
• Create data tools for analytics and data scientist team members that assist them in building and optimizing our product into an innovative industry leader.
• Work with data and analytics experts to strive for greater functionality in our data systems.
• Design and architect solutions with Big Data technologies (e.g Hadoop, Hive, Spark, Kafka)
• Design and implement systems that run at scale leveraging containerized deployments
• Design, build, and scale data pipelines across a variety of source systems and streams (internal, third-party, as well as cloud-based), distributed/elastic environments, and downstream applications and/or self-service solutions

Minimum Required Qualifications:

• Bachelor’s Degree in Computer Science, Computer Engineering, Informatics, Information Systems, or another quantitative field
• Minimum 5 years of experience in a Data Engineer role

Required Skills:

• Experience with big data tools: Hadoop, Spark, etc.
• Experience with relational SQL and NoSQL databases, including Postgres
• Experience with AWS cloud or remote services: EC2, EMR, RDS, Redshift
• Experience with stream-processing systems: Kafka, Storm, Spark-Streaming, etc.
• Experience with object-oriented/object function scripting languages: Python, Julia, Java, C++, Scala, etc.
• Experience with data encryption/security features applied to data-in-transit
• Advanced working SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases
• Experience building and optimizing ‘big data’ data pipelines, architectures and data sets
• Experience performing root cause analysis on internal and external data and processes to answer specific questions and identify opportunities for improvement
• Experience with development, management, and manipulation of large, complex datasets
• Experience with database & ETL technologies
• Demonstrated knowledge of data management competencies and implementation
• Understands the Big Data related problems and requirements to identify the correct technical approach. Core understanding of Big Data principles and architectural patterns
• Working knowledge of message queuing, stream processing, and highly scalable ‘big data’ data stores
• Experience supporting and working with cross-functional teams in a dynamic environment

Preferred Skills:

• Experience with containerized deployment technologies (Kubernetes, Openshift, etc.)
• Experience with instantiating and configuring Virtual Machines (VMware, VirtualBox, etc.)
• Experience with Machine Learning algorithms and applications interfacing with data management solutions

Competencies:

• Solid analytical abilities, coupled with a strong sense of ownership, urgency, and drive
• Attention to detail
• Initiative, creativity, reliability, teamwork
• Ability to adapt to and thrive in a fast-paced environment

EEO Statement:IDT is an Equal Opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, creed, sex, sexual orientation, gender identity, national origin, disability, protected Veteran status, or any other basis protected by federal, state, or local law.