Data Engineer

Description

Wiliot was founded by the team that invented one of the technologies at the heart of 5G. Their next vision was to develop an IoT sticker, a computing element that can power itself by harvesting radio frequency energy, bringing connectivity and intelligence to everyday products and packaging, things previously disconnect from the IoT. This revolutionary mixture of cloud and semiconductor technology is being used by some of the world’s largest consumer, retail, food and pharmaceutical companies to change the way we make, distribute, sell, use and recycle products.

Our investors include Softbank, Amazon, Alibaba, Verizon, NTT DoCoMo, Qualcomm and PepsiCo.

We are growing fast and need people that want to be part of the journey, commercializing Sensing as a Service and enabling “Intelligence for Everyday Thing”.

Wiliot is seeking an experienced Data Engineer to join our team in one of our key locations: San Francisco, New York, or Dallas. This role will focus on developing and deploying production pipelines in addition to accelerating development of Wiliot’s product for strategic customers by optimizing data tools and infastructure.

Responsibilities

• Data Pipeline Development: Design, build, and maintain scalable data pipelines using CI/CD and Git to incorporate best practices. This includes implementing ETL/ELT processes of IoT data to ensure prompt access to business-ready data.
• Tool Development for Data Scientists: Create and refine tools and frameworks that enable data scientists and analysts to efficiently query, process, and visualize data. This includes custom scripting, development of automated data quality checks, and creation of data models that align with business requirements.
• Data Architecture: Work on the architecture of our data systems, ensuring they are robust, scalable, and secure. This involves optimizing data flow for real-time analytics and machine learning model training.
• Integration & Interoperability: Integrate various data sources and ensure interoperability between various cloud services and microservices
• Performance Optimization: Monitor and enhance the performance of data systems, including tuning SQL queries, optimizing data ingestion rates, and managing data warehouse performance.
• Collaboration: Work closely with data scientists, analysts, and engineers to understand their needs and provide tailored data solutions. Participate in cross-functional teams to drive projects from conception to deployment.
• Documentation and Knowledge Sharing: Maintain comprehensive documentation of data pipelines, tools, and processes. Mentor junior team members and share knowledge to foster a culture of continuous learning.
• Staying Current: Keep abreast of new developments in IoT, big data technologies, and data engineering practices to continuously improve our data infrastructure.

Requirements

Education:

Bachelor's or master’s degree in computer science, Machine Learning, or a related field.

Experience:

• 3-5 years of experience in data engineering roles with at least one role in a technology company.
• Proven experience in Apache Spark and Python
• Experience deploying containerized applications via Docker, Kubernetes, and/or Terraform

Technical Skills:

• Proficient in programming languages like Python.
• Experience with big data technologies such as Apache Spark, Kafka, or similar tools.
• Experience implementing streaming applications with Apache Kafka, Spark Structured Streaming, or Flink
• Knowledge of cloud platforms (AWS, GCP, Azure) with practical experience in implementing cloud-based data solutions.
• Familiarity with database systems, both relational (e.g., PostgreSQL) and NoSQL (e.g., MongoDB), as well as data storage formats such as Parquet and Delta.
• Expertise in version controls systems such as Git

Additional Skills:

• Experience with Databricks and workflow management tools such as Databricks Workflows or Apache Airflow
• Expertise with build pipelines such as Github Actions or Bitbucket Pipelines preferred
• Proficiency with Java and/or Scala
• Knowledge of code design frameworks such as microservices, domain driven design, functional programming, and event-based application design
• Strong analytical and problem-solving skills.
• Excellent communication skills to liaise between technical and non-technical stakeholders.
• Ability to independently manage and progress on multiple projects simultaneously.

What We Offer:

• Competitive salary and benefits package.
• Opportunities for professional growth in a dynamic and innovative IoT environment.
• A collaborative, inclusive work culture with a focus on cutting-edge technology solutions.

#LI-Hybrid