SpeakEasy AR

Inspiration

Public speaking anxiety is something a lot of people deal with, especially when they start speaking too fast without noticing it. We wanted to build something that could support speakers in a subtle way instead of distracting them. That idea led us to SpeakEasy AR and its PaceCoach feature.

What it does

SpeakEasy AR gives speakers lightweight support directly in AR while they talk. It shows rolling live captions and tracks speaking pace in real time, including both immediate and more stable words-per-minute estimates. The goal is to help users notice whether they are rushing, slowing down too much, or staying at a comfortable pace.

How we built it

We built the core system in Lens Studio using Spectacles’ native AsrModule and custom TypeScript scripts. On the back end, we created a continuous speech recognition pipeline, a rolling caption system, and a pace-tracking engine based on sliding windows and live transcript updates. On the front end, we designed AR text overlays for captions and feedback so the information stays visible but lightweight.

Challenges we ran into

One of the biggest challenges was making continuous ASR feel stable in real use. We had to handle duplicate transcript updates, session timing issues, and network-related interruptions. We also ran into a lot of UI debugging because Lens Studio Preview and the actual Spectacles display behaved differently, especially for screen placement and layout.

Accomplishments that we're proud of

We are proud that we got the core speech pipeline working end to end on Spectacles. The system can continuously capture speech, display live rolling captions, and estimate pace in a way that is useful for speaker feedback. We are also happy that we built both the underlying logic and a working AR interface instead of stopping at just a prototype concept.

What we learned

We learned that building a smooth AR experience is not only about the model or the algorithm, but also about timing, clarity, and placement. Small issues like transcript duplication, delayed updates, or badly placed UI can affect the whole experience. Most importantly, we learned that helpful AR should stay in the background and support the user without overwhelming them.

What's next for SpeakEasy AR

Next, we want to make the feedback smarter and more polished. That includes improving pace accuracy, adding stronger visual metrics for speech patterns, and making the overlay feel cleaner and more adaptive during longer speeches. We also want to expand beyond pacing so SpeakEasy AR can become a broader speaking support tool for confidence, clarity, and communication.

Built With

• javascript
• snapspectacles
• spectaclesinteractionkit
• typescript