Software Engineer - Big Data Ingestion and Processing

About the OrganizationNow is a great time to join Redhorse Corporation. Redhorse specializes in developing and implementing creative strategies and solutions with private, state, and federal customers in the areas of cultural and environmental resources services, climate and energy change, information technology, and intelligence services. We are hiring creative, motivated, and talented people with a passion for doing what's right, what's smart, and what works.
About the RoleRedhorse is transforming how government agencies leverage data and technology. We are seeking a highly skilled Software Engineer to join our team supporting a critical intelligence mission. You will play a vital role in ingesting, processing, and analyzing massive datasets, directly impacting the Sponsor's ability to address pressing intelligence questions. You will work with cutting-edge technologies in a dynamic, collaborative environment, directly contributing to national security.

Key Responsibilities

• Load large datasets into the Sponsor’s on-premises and Cloud environments.
• Develop and maintain ingestion algorithms and schemas for large datasets.
• Analyze new large-volume datasets to optimize the data ingest processes.
• Support the creation of Apache NiFi schemas for new data loads.
• Develop software tools that efficiently preprocess, modify, aggregate, load, index, and archive large data collections into clusters in near real-time.
• Ensure proper access controls are implemented.
• Generate metrics to track data ingest statistics to maintain data integrity and provenance.
• Document the data-flows according to standards set by the Sponsor.
• Engage regularly with data scientists, analysts, and managers.

Required Experience/Clearance

• Demonstrated professional experience in Computer Science, Computer Engineering, Systems Engineering, or closely related discipline.
• Demonstrated professional experience with AWS cloud services, including long-term storage options, and cloud-based database services.
• Demonstrated experience working with Databricks.
• Demonstrated experience understanding SQL database structures and mapping them between different SQL databases.
• Demonstrated professional experience working with Apache NiFi.
• Demonstrated professional experience working with large data and high-performance compute clusters such as Hadoop or similar.
• Demonstrated experience with API development techniques.
• Demonstrated experience developing and deploying ETL processes for large data sets.
• Demonstrated experience creating operating system level scripts to perform ETL operations on SQL databases.
• Demonstrated professional experience with version control systems, preferably Git.
• Demonstrated experience testing the development of software solutions for the extraction, transformation, and loading of data using the most efficient languages for the task such as NiFi, Python, and SQL.
• Demonstrated experience implementing multiprocessing data-flows to parallelize ingest operations.
• Minimum 5-7 years of relevant experience.

Desired Experience

• Demonstrated experience with the Sponsor’s data environment.
• Demonstrated experience exhibiting strong coordination and collaboration skills.
• Demonstrated experience working with full-stack developers to deploy applications that leverage large data sets.
• Demonstrated experience communicating technical concepts to non-technical audiences.

Equal Opportunity Employer/Veterans/Disabled  Accommodations:If you are a qualified individual with a disability or a disabled veteran, you may request a reasonable accommodation if you are unable or limited in your ability to access job openings or apply for a job on this site as a result of your disability. You can request reasonable accommodations by contacting Talent Acquisition at Talent-Acquisition@redhorsecorp.com Redhorse Corporation shall, in its discretion, modify or adjust the position to meet Redhorse’s changing needs.This job description is not a contract and may be adjusted as deemed appropriate in Redhorse’s sole discretion.