Master Thesis - Finding and solving software build issues using LLMs

Job Title

Master Thesis - Finding and solving software build issues using LLMs

Job Description

Category

LLM, CI, Software Toolchains, Build Systems 

Scope

2 students completing 22,5 or 30 credits each. Scope can be varied. 

Background

At Axis Communications, our continuous integration (CI) chain executes tens of thousands of builds daily. Each build process generates extensive logs during various stages such as compiling, unit testing, integration testing, and static analysis. Every code change committed to our firmware codebase passes through this rigorous CI pipeline. 

Identifying and isolating errors within these massive logs is a tedious and time-consuming task for engineers. To alleviate this, we have developed a "Build Failure Analyzer" service that scans logs for known patterns, classifying errors to enhance navigation through the logs. However, maintaining these handcrafted patterns is labor-intensive, and as our software evolves, new and unseen errors emerge that the analyzer cannot detect. Engineers then face the daunting task of sifting through thousands of log lines to pinpoint issues. 

Goal

We propose leveraging Large Language Models (LLMs) like Llama or Mistral to automate the analysis of build logs and identification of errors, removing the need to handcraft error patterns all together. 

Main Objectives 

• Build an annotated dataset to help evaluating the various LLMs, log data consuming methods, parameters etc 

• Evaluate the model's ability to detect previously unseen errors by leveraging the LLM's understanding of language patterns. 

• Test the model on logs with injected synthetic errors to simulate new error scenarios. 

• Propose strategies for integrating the LLM into our existing CI pipeline and developer tools for real-time error detection. 

• Conduct user studies to gather feedback from engineers on the tool's utility. 

• Define metrics such as precision, recall, and time savings to evaluate the model's effectiveness compared to the current system. 

Prompt Engineering and Input Formatting: 

• Experiment with various input representations to optimize the LLM's focus on critical log sections. 

• Develop prompts that elicit more accurate and concise error identifications from the model.

Explainable AI and Interpretability: 

• Implement methods to make the model's feedback intuitive for the user, such as highlighting relevant log parts. 

Expected Outcomes: 

• A robust LLM-based tool capable of automatically detecting and classifying errors in build logs with high accuracy. 

• A comprehensive evaluation of the LLM's performance compared to the existing pattern-based system (Build Failure Analyzer), highlighting strengths and limitations. 

• Guidelines and best practices for integrating LLMs into industrial CI pipelines. 

• Insights into the scalability and adaptability of LLMs for log analysis across different projects and domains. 

• Recommendations for future enhancements and potential extensions of this approach. 

Who are you? 

Most likely you are studying a Master Program within Engineering and are interested in the areas stated above. 

OK, I am interested! What do I do now?
You are valuable to us – how nice that you are interested in one of our proposals! There are a few things for you to keep in mind when applying.

• Applications are accepted in both Swedish and English, and you apply via the proposal advert.
• The announced thesis is open only to students affiliated with a Swedish University/College either directly or via an exchange program.
• When the thesis proposal states that it includes two students working together, we would like you to apply in pairs. In these cases, send one application each but make sure to clearly state in your application who your co-applicant is. If you have any questions regarding this, please do not hesitate to contact us.
• Please attach your CV and University/College grade summary.

Who to contact for any questions regarding the position!

Please contact  Gustaf Lundh, Expert Engineer, at 0736-510968 or e-mail gustaf.lundh(at) axis.com

Type of Employment

Temporary Employment (Fixed Term)

Posting End Date

2025-02-09

Certain roles at Axis require background checks, which means applicable verifications will be done in these recruitments. Notice will be provided before we take any action.

About Axis Communications

We enable a smarter, safer world by creating innovative solutions for improving security and business performance. As a network technology company and industry leader, we offer solutions in video surveillance, access control, intercom, and audio systems, enhanced by intelligent analytics applications.

With around 4500 committed employees in over 50 countries, we collaborate with partners worldwide. Together, we thrive in our friendly, open, and collaborative culture and inspire each other to think beyond the expected. United by our commitment to inclusion, diversity, and sustainability, we consistently seek to develop our skills and way of working.

Let´s create a smarter, safer world

For more information about Axis, please visit our website www.axis.com.

Listen to Get To Know Axis – Podcast