• 

  Application Thumbnail

• 

  Call to Action with Molly the Molycule

Inspiration

We were inspired by conversations with chemistry teachers who explained their daily struggles when teaching abstract concepts like molecule reactions. Often they are challenging to understand. We tried to think of a way to represent the school experiments both in the real and the virtual world combined. We want to make it possible to actually "dive into" the molecular level to understand what is happening in detail. During this, our AI helper Molly guides the learner through the experience and answers to important questions dependent on what is currently going on. That's how we envisioned the following scenario.

What it does

The User Experience

Imagine you want to do a chemical experiment at school. Usually, the teacher sets up the experiment, demonstrates the reaction and afterward he or she speaks about it with the students or pupil. With Molly, the Molecule you can:

• make every pupil and student not only watch but participate actively in an own experiment
• allow them to dive into the molecular level and see and make the reactions happen on a microscopic level - They don't only see what's happening, they are an active part of it and immersed in the experience!
• help the atoms react with each other
• have a kind and intelligent molecule helping them solve problems and navigate the scene
• get answers to questions that may come up during the exploration of the process

Core Features

• Lets school children place a chemical experiment on a real table
• Start the experiment by pouring water in it
• dive into the molecule level
• interact with molecules, make them react and observe reactions happening all around us
• interact with an AI guide to ask questions
• get instructions through the scenes with the AI guide

How we built it

Planning and Conceptualization

• We began with a brainstorming session in ShapesXR where Adriane presented her idea on multiple slides with different states of reactions of the atoms on molecular level
• we could all already envision the application because of the MR functionality of ShapesXR
• we identified specific problems that could be addressed through an AI guide in the scene
• we decided to only focus on one experiment for now and make the whole application be flexible or expandable with more experiments for schools in the future

Design Phase

• We had a team lead for the idea in Adriane who told everyone what is important from the didactics perspective.
• Spandana and Tiago were the creative department. Spandana developed the static scene and prototyping and made the prefabs for the molecules. She iterated on multiple possibilities to display the molecules. Tiago did the user interface and graphical assets and also managed in between the departments for support and advice.

Development Process

• Jonas and David were the heads for development in Unity
• Jonas mainly implemented the AI Interactions for Molly with Whisper AI, TTS AI, and GPTo4 API and made the scene assembly and transitions possible through a statemachine
• David did the molecule behavior and interaction and implemented the scene understanding through the Meta SDKs
• We used the MRUK to understand our start scene at the beginning to place the experiment in the real world

Challenges we ran into

• Trying to complete a full chemistry experience lesson from start to finish (getting the reactions to actually perform chemically correct and meaningful from a didactics perspective is not easy!)
• AI implementation and configuration to guides through the experience and answer questions to children was a challenge as we had to understand not only the user question and intent but also understand the current scene state and prohibit questions which are not supposed to be answered during the experiment
• We ran into problems converting the Meta UI VR samples into MR
• Making molecules behave in a correct manner on their own to see their reactions around us and at the same time make it possible to interact with them was a challenge because then we couldn't just use static animations

Accomplishments that we're proud of

• Trying to complete a full chemistry experience lesson from start to finish
• Building something that provides some didactic value which would not be possible without using Augmented Reality, Virtual Reality and an AI agent helping us.
• Have a cool AI mentor that kids can ask questions and guide them through the project (Molly can listen, understand, and speak!)

What we learned

• Chemistry is fun but hard!...But that was kind of the point.

---

What's next for 16 - Molly the Molecule: The Acid Base Adventure

Short-term highest priority: We want to work on the UI and make the interaction with Molly the Molecule more intuitive and responsive

Short-term lower priority: We want to make the session collaborative and add multiplayer so that multiple students can work on one experiment together.

Long-term highest priority: We want to build chemistry lessons with more experiments and more molecules to interact with.

Long-term lower priority: We think of having a community plattform in the future for questions where kids can get inspired and teachers learn how their colleagues work with the application.

---

Walkthrough Video: https://youtu.be/HsrHIAMcn70?si=wISiifdQovhYPlat

Built With

• aipowered
• blender
• chatgptapi
• handinteraction
• mruk
• openai
• passthrough
• pearson
• pixyzstudio
• roomsetup
• sceneanchors
• shapesxr
• tts
• unity
• whisper
• winner