An integrated LLM which interprets and responds to user actions and elements in an educational VR environment

High level description

This thesis will be written in collaboration with Knightec and ByBrick.

ByBrick is a digital agency with more than 15 years of experience within creating interactive 3D-applications and simulations for VR, AR, mixed reality and the web. Their current focus in VR is building VR-simulations for training purposes.

One of the key benefits of VR in education is its ability to provide students with hands-on experiences and simulations. It allows them to explore and interact with complex concepts, environments, and scenarios that could be critical or even dangerous.

VR, in combination with LLMs, are used to create immersive learning experiences that combine virtual and physical environments. LLMs are used to both evaluate and provide feedback on responses to open-ended questions.

The next step for LLM in VR would be to not only take user questions as input but also consider the VR environment and what physical choices and actions the user has executed. Which would result in an AI-coach that not only hears questions but also “sees” the needs.

Project description

To further help and support a student in VR education we propose an LLM based AI-coach. The AI-coach would use the actions from the user in VR as input to help the user with hints and explanations to what the user might have done wrong.

An existing VR – education in electrical circuit troubleshooting will be used for this thesis.

The students will:

• Research the possibilities of how a LLM can interpret actions and components from a VR environment. Can both images and text be used as prompts for the LLM?

• Analyze which type of LLM is more suitable for this task. Can an existing LLM be used or is modification required?

What are we looking for

We are looking for a pair of master’s students with an interest in machine learning, artificial intelligence, virtual reality and electronics.

Purpose

The purpose of this master thesis is to analyze and find a solution on how to train an LLM to respond to VR actions in an electronics educational environment.

The thesis project can be published and used in your personal portfolio as well as in company marketing. Include Resumé/CV and cover letter in your application.