LinguaLens

Inspiration

I practice English every day by talking with GPT-Live. And every day, the same thing happens: mid-sentence, I get stuck. The phrase I want is right there, but it won't come out. Looking at my phone kills the conversation. By the time we finish talking, I've even forgotten what I was stuck on. The best learning moment of my day evaporates — every single day.

LinguaLens was built to catch exactly that moment, without breaking the conversation.

What it does

LinguaLens runs on Even Realities G2 smart glasses. While you speak — with a person, or with GPT-Live — it listens through the glasses' microphone and shows small coaching cards on the HUD only when they help:

• HINT — when you stall, 3 complete, ready-to-say phrases appear, each with a short gloss in your native language. They differ in intent (ask for time / state the blocker / propose an alternative), so you pick one and say it. The conversation never stops.
• WORD — when the other side uses a difficult word, a plain-English gloss or a short example sentence appears.
• RECAP — in a quiet moment, the phrase you couldn't say comes back as a review card, and again at your next session. The freeze becomes the lesson.
• Tap to ask — a single temple tap requests help on demand, so you are never dependent on detection alone.

The core design value is silence: the model is explicitly allowed to answer "no intervention." A good coach knows when to stay quiet.

How we built it

• Every line of code was written through Codex CLI sessions (session IDs available; the timeline is in the git history). An orchestrator handled specs, review, and simulator verification; Codex implemented.
• GPT-5.6 (sol) generates RECAP phrases — low-frequency, quality-critical. gpt-5.6-luna makes the high-frequency intervene-or-stay-quiet call and, crucially, attributes who said what. gpt-4o-mini-transcribe handles ASR from the glasses' 16 kHz mic, gated by client-side VAD so silent audio never hits the API. A built-in $1/day budget guard halts live calls at the cap.
• BLE bandwidth is the design constraint: the link sustains roughly 10–30 KB/s, so image animation is impossible (~1 fps). LinguaLens is designed backwards from that: text-first cards, one hero image under 200×100 px at ≥1 s intervals, and a single 250 ms heartbeat driving all timing.

Challenges we ran into

• You cannot tell who is speaking. The G2's 4-mic array is exposed as one merged stream — no diarization, no direction of arrival. Our first build guessed roles with keyword regex, and it was wrong often enough to make the coaching feel random: say "Could we push the deadline?" yourself and it was attributed to your partner. The fix was to stop guessing locally and let the model attribute roles from conversational context, which it does well because a fluent partner and a stalling learner sound structurally different.
• Timing was worse than wording. Running a judgment on every 3.2 s audio chunk interrupted mid-sentence and missed real stalls. We replaced it with stall detection — filler words, a sentence trailing off on a preposition, silence after a question — so the model is consulted only at moments that matter. Cost dropped with it.
• The strongest signal was being thrown away. When a Japanese learner freezes, they leak Japanese ("えっと、締切延ばしたい…"). That is the intent. We now treat those fragments as the primary source of intent rather than inferring from English context, and hints continue the sentence you already started instead of replacing it.
• The simulator virtualizes WebView timers, silently freezing chained setTimeouts; all timing was rebuilt on one elapsed-time heartbeat.
• Glasses plugins are foreground-only with no audio output at all — which validated the product: a visual, silent coach is the only kind that can exist here.

Accomplishments that we're proud of

• A complete conversation-coaching loop (mic → ASR → stall detection → judgment → HUD → review) inside a ~48 kbps sustained budget.
• Judges can run the full pipeline in ~2 minutes with no hardware and no API key: the official desktop simulator plays a scripted conversation with the whole loop live.
• The G2 IMU's axes are undocumented ("TBD" in the official FAQ); our flag-guarded nod-gesture experiment ships with a safe fallback.
• On-device session telemetry records what was said, what was suggested, and when — so tuning is driven by data, not impressions. Logs stay on the device unless you configure an endpoint.

What we learned

Restraint is a feature, and the hard part of an in-conversation assistant is not generation — it is knowing when. On a 576×288 green display with a thin radio link, every byte and every second of attention must be earned, and that pressure produced a better product than a bigger canvas would have.

What's next for LinguaLens

• GPT-Live practice mode: GPT-Live does the talking, LinguaLens does the catching — the same HUD in practice and in real life (waiting on the GPT-Live API).
• gpt-realtime streaming to cut hint latency to sub-second.
• Native-language glosses beyond Japanese via the SDK's user-country signal.

Built With

• codex
• even-hub-sdk
• gpt-4o-mini-transcribe
• gpt-5.6
• gpt-5.6-luna
• typescript