SpeakSee

Inspiration

We were inspired by the challenges English learners face with pronunciation. It’s incredibly difficult to connect the sound of a word with its written form, and traditional apps keep you staring at a phone screen. We saw an opportunity with Snap Spectacles to create an immersive, heads-up learning tool. We wanted to build something that lets users practice in a more natural, hands-free way, turning their real environment into a language lab.

What it does

SpeakSee turns your world into an immersive English classroom. It's an AR lens for Snap Spectacles that directly tackles pronunciation. When a user hears a spoken English phrase through the Spectacles, SpeakSee instantly displays the matching subtitles in their field of vision. This hands-free experience lets users see, hear, and connect the sound of a word with its written form in real-time, allowing them to practice their pronunciation anywhere.

How we built it

Challenges we ran into

Audio-Text Synchronization: The biggest challenge was perfectly timing the subtitles to the audio. We had to fine-tune the script's timing so that the text appeared exactly when the corresponding words were spoken, which was crucial for the tool to be effective.

Modifying Sample Code: Understanding and integrating our code into the pre-existing sample project was a challenge. We had to be careful to add our subtitle feature without breaking the original voice playback functionality.

AR User Interface: Designing for AR is very different from a 2D screen. Figuring out the best font size, color, and placement for the subtitles so they were readable against any real-world background, but not distracting, took several attempts.

Accomplishments that we're proud of

Our biggest accomplishment is the seamless, real-time synchronization between the audio playback and the AR subtitles. Getting the text to appear exactly when the corresponding words are spoken was a significant technical challenge, and it's the core feature that makes the tool feel so intuitive and effective. We're proud of successfully modifying a complex sample project to build a practical, user-focused learning tool, not just a tech demo.

What we learned

What's next for SpeakSee

This is just the beginning. Our next steps are focused on making SpeakSee a fully interactive tutor:

Voice Recognition & Feedback: The top priority is to add speech-to-text. This will allow users to repeat the phrase they hear, and SpeakSee will "listen" and provide real-time feedback on their pronunciation accuracy.

Expanded Content Library: We plan to build a much larger library of words and phrases, allowing users to select categories based on difficulty or real-world situations (like "Ordering Food" or "Job Interview").

Multi-Language Support: The framework we built is language-agnostic. We aim to expand SpeakSee to help users learn many other languages, not just English.