Analytics - Software Engineer

 

Responsibilities What you'll do… Engineer, test, document and manage GCP Dataproc, DataFlow and VertexAI services used in high-performance data processing pipelines and Machine Learning. Help developers optimize data processing jobs using Spark, Python, and Java. Collaborate with development teams to integrate data processing pipelines with other cloud services and applications. Utilize Terraform and Tekton for infrastructure as code (IaC) and CI/CD pipelines, ensuring efficient deployment and management of cloud resources. Monitor VertexAI, DataFlow and Dataproc services infrastructure, identifying and resolving issues to ensure high availability and minimal downtime. Support different teams in troubleshooting VertexAI setup, DataFlow jobs, Dataproc jobs and Dataproc cluster issues Stay up-to-date with the latest cloud technologies and best practices, continuously improving our cloud infrastructure and services.

 

Qualifications You'll have... Bachelor’s degree in Computer Science, Engineering, or a related field. 2+ years of experience supporting Big Data and Analytics platforms Experience with GCP infrastructure and services, specifically VertexAI, Dataproc and DataFlow. Excellent problem-solving and analytical skills. Strong communication, documentation and collaboration abilities. Ability to work in a fast-paced, dynamic environment. Development experience with any programming or scripting languages preferably Python or Java.

 

Even better, you may have... Experience with Spark for large-scale data processing. Solid understanding and experience with GitHub for version control and collaboration. Experience with Terraform for infrastructure management and Tekton for continuous integration and deployment. Experience with Apache NiFi for data flow automation. Knowledge of Apache Kafka for real-time data streaming. Familiarity with Google Cloud Pub/Sub for event-driven systems and messaging. Familiarity with Google BigQuery for data analytics and warehousing solutions. Practical experience with Docker for containerization and deployment. Experience with Google Cloud Run for managing and deploying containerized applications. Experience with Apache Beam for building robust, scalable data processing pipelines. Experience with Apache Airflow for orchestrating complex computational workflows and data processing pipelines.