Data Platform Engineering Team Lead (Hands-On | Data Lake Focus)

Description

Modern vehicles create an enormous amount of data. Upstream turns that data into real-time cybersecurity detection & response (XDR), API security, and proactive quality detection for the world’s largest automotive and smart mobility companies. Our Data Engineering & Data Science Group leads the development of our Iceberg-based data platform, including data lake, query engine, and ML-Ops tools, serving as a solid AI-ready foundation for all our products.

At Upstream, you’ll have real impact, your work powers every security alert and analytics view we deliver. You’ll have room to innovate, pick new tools, refactor old ones, and set best practices. All within a people-first culture that values hybrid work, shared knowledge, and sustainable pace.

Technological background: Iceberg, Trino, Prefect ,GitHub Actions, Kubernetes. JupyterHub, MLflow, dbt

This role is full time and is Herzliya, Israel based.

Responsibilities

• Spend about 70% of your time coding, building and tuning the data lake (Iceberg + Trino + dbt) for fast, cost-effective queries.
• Lead and mentor skilled data engineers: code reviews, pairing, growth plans, and future hiring.
• Own the platform roadmap including schema design, query-engine upgrades, governance and new technology choices.
• Design and maintain well-partitioned tables and views so DS, DA, users and product teams can work faster and effectively.
• Automate ETL data-digests and ML model training with Prefect, and own the pipelines for deployments (GitHub Actions → Kubernetes), data-quality checks, monitoring and SLOs.
• Enhance and scale our ML platform, and automate the machine-learning lifecycle (MLflow, feature stores, model builds and deployment).
• Stay focused on the data lake.

Requirements

• 8+ years of software/data engineering (strong Python preferred; JVM/Go welcome).
• 4+ years working with large-scale data lakes/lakehouses, including Iceberg (or similar formats). Experience with Apache Spark for heavy batch / ML pipelines is a plus.
• 2+ years leading engineers, while still being hands-on.
• Solid SQL skills and experience with much of our stack: Trino/Presto, dbt, Iceberg, Prefect/Airflow, Kubernetes, GitHub Actions.
• A builder’s mindset: measure, improve, document, then automate.
• Clear communication in English and a collaborative attitude.

Upstream is an equal opportunity employer. All candidates for employment will be considered without regard to race, color, religion, sex, national origin, physical or mental disability, veteran status, or any other basis protected by applicable federal, state or local law.