GuidedByGlass

Inspiration

We wanted to explore how XR glasses could actively support communication between spoken language and sign language. Our long-term vision is a system that works in both directions: speech to sign and sign to speech, running naturally on lightweight wearable devices.

⸻

What it does

GuidedByGlass is a prototype that listens to speech on Android and displays the corresponding sign visually. Alongside the Android version, we experimented with showing our own 3D signing hand models inside both XReal glasses and Snap Spectacles. These models can be paired with short text cues to make the gestures easier to follow.

⸻

How we built it

We built an Android app with speech recognition that displays matching sign images. In parallel, we explored integrating this concept into XReal glasses by adapting the Android workflow for XR. We also created 3D signing hands using Meshy AI and refined them in Blender. These models were imported into Lens Studio to test how they behave inside the Snap Spectacles environment.

⸻

Challenges we ran into

Each device had an entirely different development pipeline, and none of us had XR experience before the hackathon. Preparing Blender exports for XReal was difficult because of missing files and format issues. Lens Studio introduced another learning curve with its constraints and limited documentation. Switching between Android, XReal, and Snap while also generating and refining 3D assets made the process technically intense under time pressure.

⸻

Accomplishments we’re proud of

We managed to build functioning prototypes on two different XR devices and created a working Android app from scratch. We produced our own AI-generated signing hand models, brought them into XR, and implemented the first version of speech-to-sign translation. Given our limited XR background, we are proud that we were able to design, model, integrate, and test across multiple platforms.

⸻

What we learned

We learned Android development, how to work with the XReal pipeline, how Unity-style XR workflows differ across devices, and how Lens Studio scripts and interactions work. We also learned how to use Meshy AI to generate models, how to refine them in Blender, and how to import them into XR.

⸻

What’s next for GuidedByGlass

We want to build the reverse pipeline and enable sign-to-speech through gesture recognition. We aim to expand the vocabulary, improve our 3D sign models for Snap, and bring the full experience to the XReal glasses. Our goal is to create a natural, hands-free translation tool that works on any XR device and supports communication in both directions.

Built With

• javascript
• kotlin