Thesis Work 30hp - Merging Code Large Language Models for Enhanced Code Generation

Thesis projects at Scania are excellent ways of making contacts for your future working life. Many of our current employees started their career with a thesis project.

Background:
Large Language Models (LLMs) have revolutionized code generation, enabling developers to write more efficient and accurate software with the assistance of AI. However, individual models often have limitations in scope, specialization, or performance. Model merging—combining multiple LLMs to leverage their unique strengths—presents a promising avenue to create superior models without the high computational costs associated with fine-tuning. By merging code-specific LLMs, we aim to enhance their capabilities, achieving better performance on diverse coding tasks while maintaining cost-effectiveness.

Target:
This project explores the potential of merging multiple code-focused LLMs to develop a unified model that excels in various code generation tasks. The research will investigate different merging techniques, evaluate their effectiveness, and identify the optimal strategies for combining models to maximize performance. Potential contributions include improved code generation accuracy, increased versatility across programming languages, and reduced reliance on expensive computational resources. Additionally, the project will examine the theoretical and practical challenges of model merging, such as compatibility of model architectures and the preservation of specialized knowledge.

Example of assignments:
•    Develop Merging Algorithms – Study and or develop different algorithms for merging multiple code LLMs.
•    Evaluate Performance – Compare the merged model against individual models and fine-tuned alternatives on various code generation benchmarks to assess improvements in accuracy and efficiency.
•    Develop Benchmark – Develop novel code generation benchmark.

Education:
MSc in Computer Science or similar, with some background in formal methods.

Contact person and supervisor:
Liv Kåreborn, AI Sweden, liv.kareborn@ai.se
Minal Patil, senior researcher, Scania, minal.patil@scania.com
Mattias Nyberg, Adj. prof, KTH / Research Lead, Scania,  mattias.nyberg@scania.com

Number of students: 1-3
Time:20 weeks, full time 40 hours per week
Start: Jan 2025
Credits: 30hp
 
Application:
Enclose CV, personal letter and transcript of grades.
Application shall be registered in both: Thesis project application, and the "Apply"-button on this page
 
A background check might be conducted for this position. We are conducting interviews continuously and may close the recruitment earlier than the date specified.