Master's thesis: Integer Transformer via Lottery Ticket Hypothesis

Background. Large Language Models (LLMs) are powerful but computationally demanding, especially when deployed in resource-constrained environments. This project is hosted by RISE Research Institutes of Sweden, a state-owned research institute that supports sustainable innovation across academia, industry, and the public sector. The project explores the intersection of efficient neural architectures and quantization techniques to enable integer-only Transformers.

Description. This thesis investigates the development of an integer-only Transformer model by combining the Inhibitor attention mechanism—which uses Manhattan distance and ReLU for efficient integer arithmetic—with the Neural Networks Lottery Ticket Hypothesis. The goal is to identify sparse, trainable sub-networks with integer weights and evaluate their performance on standard NLP benchmarks.

Key Responsibilities

• Conduct a literature review on quantization, sparse training, and the Lottery Ticket Hypothesis

• Implement the Inhibitor Transformer with integer-only weights

• Train and evaluate smaller models on tasks such as sentiment classification and knowledge distillation

• Benchmark computational efficiency and standard NLP task performance

• Document findings in a scientific report

Qualifications

• Strong background in mathematics, statistics and deep neural networks

• Familiarity with model compression, quantization, or sparse training techniques

• Proficiency in Python and frameworks like PyTorch or TensorFlow

• Experience with NLP tasks and datasets is a plus; cryptography likewise

Terms

• Scope: 30 hp, one semester full-time

• Location: Luleå (or remote with agreement)

• Start: Flexible

• Compensation: 10,000 SEK for travel, materials and the like after the project is completed and approved

Please note: You need to have a valid student visa that allows you to study in Sweden during the thesis period.

Welcome with your application

Last day of application: July 29

Contact: Rickard Brännvall (rickard.brannvall@ri.se), Dilletta Romano, Joakim Eriksson

Check-in questions (yes/no): 1-5 are required, 6-9 are beneficial, 10 and 11 are specifically plus