Inspiration

We were inspired by Apple Vision Pro. We asked, what if an astronaut had that same kind of assistant, one they could simply talk to when no one else is around to help?

What it does

Nova is a hands-free AI. Say “Hey Nova” and it answers questions out loud, monitors you sensors, displays a minimap of the terrain, and guides you to waypoints. Our current build has a Mars simulation for testing out all of Nova's features.

How we built it

We built Nova in Unity on a custom Mars terrain, using the Quest 3 microphone for wake-word detection and speech-to-text/text-to-speech. A BERT model we trained on our own collected data identifies what the explorer wants.

Challenges we ran into

Our model often predicted inaccurately, misreading what the explorer was asking for. Improving it meant collecting more training data and retraining, all within the time limit of a hackathon.

Accomplishments that we're proud of

Nova can actually listen and answer, completing the full loop from a spoken question to a spoken reply inside the headset

What we learned

We learned what goes into building a model from start to finish: collecting and preparing the training data, training the model, and applying it inside a live app. Seeing it make mistakes in the headset showed us how much each step affects the final result.