Lead Data Engineer

Founded in 2016, Katalon is the leading provider of a modern, comprehensive quality management platform. Katalon Platform enables quality assurance, DevOps, and software teams of any size to deliver world-class customer experiences faster, easier, and more efficiently.

Since its first launch, Katalon has experienced tremendous growth, serving more than 100,000 users across 30,000 teams of all shapes & sizes, many of which are in the Fortune Global 500, such as PwC, KPMG, Abbott, etc. Katalon is recognized as a top automation tool by prestigious review sites, such as Gartner, Capterra, and IT Central Station. 

ABOUT POSITION

Katalon TrueTestis an AI-augmented test automation solution that automatically discovers, models, generates, and maintains user-journey test cases. TrueTest provides a streamlined test generation process, saving time and improving testing efficiency. As a Lead Data Engineer, you will play a critical role in developing and enhancing our flagship test automation tool, following the responsibilities of:

• Collaborate with the product manager and team to develop and enhance the product's new features in addition to utilizing Generative AI technology.
• Perform detailed technical analysis and design to break down the feature based on the high-level design and business requirements.
• Develop and review code that meets the development standards for code style, design patterns, readability, and maintainability and integrates best practices for scaling.
• Work with peer engineers to grow the technical accountability of the system to ensure release product quality, security, and performance.
• Utilize Code Pilot to increase development efficiency and quality by automating repetitive tasks and identifying potential bugs or issues early on.
• Identify areas for improvement within the existing codebase and suggest solutions to improve. 
• Identify and contribute to internal engineering working groups to define and build internal libraries, tools, and frameworks for reusability and centralize practices across
• Diagnose and troubleshoot issues to support customer requests. 
• Support automation QAs and product specialists to maintain the demo system and write scripts using new Katalon features, to test applications.

Requirements

Must-have: 

• Core Technical Expertise:
• Proficient in Python: Extensive experience with Python for batch data processing, ETL pipeline development, and large-scale data workflows.
• Apache Spark (PySpark): Expertise in distributed data processing, performance optimization, and managing Spark-based workflows.
• Apache Airflow: Proven experience orchestrating complex batch workflows and automating ETL processes using Apache Airflow.
• ETL/ELT Pipeline Design: Deep knowledge of designing, implementing, and optimizing scalable data pipelines, focusing on data transformation, validation, and performance improvements.
• Data Pipelines & Processing:
• Batch Data Pipelines: Hands-on experience developing and optimizing batch data pipelines for processing large datasets efficiently.
• Real-time Data Streaming: Expertise in building and managing real-time data streams using Apache Kafka to handle high-throughput, low-latency data.
• Structured & Unstructured Data: Proficient in handling diverse data types, including structured, unstructured (logs, text, images), and semi-structured data.
• Data Lakes & Storage: Familiar with data lake architectures (S3-based) and storage formats like Parquet, ORC, and Avro for optimized processing and storage.
• Development & Machine Learning Integration:
• Machine Learning Pipelines: Experience integrating machine learning models into data workflows to enhance automated decision-making and insights.
• CI/CD for Data Workflows: Familiarity with continuous integration and continuous deployment practices (using tools like GitHub Actions, Jenkins) for automating data pipeline workflows.
• Optimization & Scalability: Strong understanding of software engineering principles, with a focus on code optimization and building scalable data engineering solutions.
• Communication & Collaboration:
• Cross-Functional Collaboration: Ability to work effectively with data scientists, machine learning engineers, and software teams to integrate data-driven insights and AI-enhanced features.
• Problem Solving: Strong analytical and problem-solving skills with a proactive approach to identifying and addressing bottlenecks in data workflows.

Benefits

At Katalon, we bring together self-starting, open-minded, and talented people while actively promoting a transparent and growth-enabling working environment. But don’t just take our word for it. Take a better look below!

• Competitive Pay & Bonuses: We believe in rewarding great work! You'll receive an attractive salary package plus performance bonuses to help you meet your financial goals.
• Your Health & Happiness Matter: Take care of yourself with our comprehensive health coverage, flexible work options, and generous time off. We understand that life happens outside of work too!
• Location-Tailored Benefits: Enjoy a complete benefits package designed specifically for your country, giving you the best coverage where you live.
• Everything You Need to Succeed: Work with top-of-the-line equipment and enjoy modern facilities, plus helpful allowances to support your work setup.
• A Place Where You Belong: Join our worldwide family where we celebrate what makes each of us unique. Here, everyone has a voice and equal opportunities to shine.
• Room to Grow & Thrive: Your success is our success! We foster a trust-based culture where you can develop your skills, take on new challenges, and be recognized for your achievements.

Katalon is proud to be an equal-opportunity employer. We care about our people and celebrate our differences. We want to work with talented, collaborative, and innovative people. We do not discriminate in hiring or any employment decision based on race, color, religion, national origin, age, sex (including pregnancy, childbirth, or related medical conditions), marital status, ancestry, physical or mental disability, genetic information, veteran status, gender identity or expression, sexual orientation, or other characteristics protected by law.