Senior Software Engineer – Data Pipelines and Services

DTN is a global leader providing insights and analytics to our customers to feed, fuel, and protect the world.  We help people make critical business decisions that impact the agriculture, oil and gas, trading, and weather industries.

It’s a difference you’ve likely felt without even knowing it. From the food on your plate to the gas in your car to the last flight you took, chances are a DTN customer made it possible. And that’s just for starters. That’s why we say, “When our customers prosper, we all win.”

Job Description:

DTN is looking for an experienced Senior Software Engineer to join a close-knit, high-energy development team within the Eco Data organization. If you're passionate about building custom data processing pipelines at scale, working with complex geospatial datasets, and solving data transformation challenges for time-sensitive weather content, we want to hear from you! In this role, you will help design, build, and optimize data models and services that convert raw weather data into fit-for-purpose formats, supporting time-series applications. If you have a deep understanding of distributed systems, cloud-native architectures, and data engineering, this role could be a great fit.
 

Responsibilities:

• Join an international team, actively participate in daily stand-ups and scrum ceremonies, and engage with peers on Slack.

• Gain a thorough understanding of the team’s backend data services and API products.

• Use open-source software and cloud infrastructure to plan, create, test, and optimize custom data transformation pipelines that convert weather model outputs into high-performance formats.

• Collaborate with stakeholders to gather product and technical requirements, providing clarity when needed.

• End-to-End Ownership: Take charge of designing, building, and optimizing data pipelines that span from raw data ingestion to delivery, ensuring high performance, scalability, and maintainability.

• Lead asynchronous code reviews to give and receive constructive feedback.

• Create comprehensive documentation to share knowledge and support operational readiness.

• Assist with incident response and provide technical guidance during production issues.
   

Required Experience:

• Deep expertise in Python and data transformation tools, including Apache Arrow, Pandas, NumPy, and Polars.

• Experience with scientific data formats like Zarr or Parquet

• Hands-on experience building distributed, cloud-native data pipelines using AWS services (e.g., EC2, S3, SQS, SNS, EKS).

• Proficient in designing, working with, and optimizing relational databases and queries (e.g., PostgreSQL, Alembic).

• Proven experience working with event-driven architectures and scaling cloud-based systems.

• Strong expertise in building custom ETL pipelines and transforming large, time-sensitive data.

• Proficient in using data partitioning and batch processing strategies, along with efficient data storage formats, to handle complex transformations at scale.

• Experience with container-based applications, particularly with Kubernetes.

• Proficient in Infrastructure-as-Code and CI/CD practices (e.g., Terraform and GitLab).

• Experience with parallel processing, utilizing multiple cores of a machine to accelerate data processing, and asynchronous coding/concurrency techniques to maximize efficiency and throughput.

• Ability to communicate complex technical concepts in simple terms, making them understandable for a wider team.
   

Nice-to-Have Experience:

• Experience with geospatial data and time-series data models, particularly for applications involving weather data and high-volume datasets.

• Familiarity with industry-specific scientific data formats like GRIB and NetCDF.

• Experience designing, building, and supporting REST APIs using frameworks like FastAPI or Typescript/Express.

• Additional expertise in a language built for concurrency, such as Node.js.

• Experience with performance optimization and observability tools like Datadog in distributed systems.

• Experience working in the Palantir platform

This position is remote-eligible in most U.S. states.  We offer a comprehensive benefits package including flexible working hours, unlimited paid time-off, insurance (medical, dental, vision, and life), company-matching 401(k), and more. 
 

The targeted hiring base pay range for this position is between $101,250 and $135,000. DTN is a pay for performance organization, which means there is the opportunity to advance your compensation with performance over time. The actual base pay offered for this position will be dependent upon many factors, including but not limited to: prior work experience, training/education, transferable skills, business needs, internal equity and applicable laws. The targeted hiring base pay range is subject to change and may be modified in the future. This role may also be eligible for market competitive variable pay and benefits. 

#LI-REMOTE
#LI- JW1

Why DTN?

OUR VISION: To be the independent, trusted source of insights to our customers who feed, protect, and fuel the world. 

OUR MISSION: Empower our customers with intelligent and actionable insights that exceed their expectations and enable their success on a daily basis. 

OUR VALUES: Customer-Focused, Forward-Thinking, People-Centric, Solution-Oriented

We have great benefits at DTN – apply today to find out more! 

At DTN, we are an equal opportunity employer. Come join us as we help feed, fuel, and protect the world!

** DTN is an Equal Opportunity Employer Minorities/Women/Veterans/Disabled