Lyrony

Inspiration

Lyrony is live karaoke for the music already playing around you. Designed for Meta Ray-Ban Display glasses, it listens without asking the listener to search for a song, aligns to the current phrase, and shows Romanized captions with color and motion that communicate pronunciation, timing, and sustained sounds. Its first pilot supports one judge-testable English, Japanese, and Korean track, with a language-pack architecture designed to grow.

Lyrony started with my daughter. She loves an up-and-coming J-Pop band and was thrilled by the idea of seeing them in concert. Then she realized that, because she does not speak Japanese, she would be surrounded by songs she loves without being able to sing the words. What should have felt exciting suddenly felt isolating. She did not want a translation lesson in the middle of the show; she wanted to participate.

That moment exposed a larger problem. International music is more accessible than ever, but singing along still assumes that listeners can read the song's original writing system. Existing lyric apps usually make people stop, unlock a phone, identify or search for the track, and stare down at a scrolling page. That breaks the social, emotional moment of live music.

That led to a simple question: what if lyrics could meet you where the music is? The product combines the immediacy of Shazam, the visual timing language of Guitar Hero, and pronunciation guidance designed specifically for international music. Display glasses make the captions glanceable while the listener stays present with the performance and the people around them.

What it does

The user taps Start listening once and continues playing music from another phone or speaker. Lyrony:

Captures a ten-second microphone sample and displays a real input-level meter. Tries a judge-friendly three-song path by transcribing the audible vocal fragment and aligning it against timed records for the supported Japanese, Korean, and English tracks. Falls back to AudD recording recognition for other music when provider access is available. Retrieves the relevant timed lyric window at runtime. Uses GPT-5.6 to create singable Romanization, a concise meaning, and sustained-syllable guidance for Japanese and Korean. Renders the phrase in Sing, Learn, or Feel mode with a moving color fill that shows how long to hold each phrase. The public site includes a full glasses simulator, cute test-song hints, honest provider diagnostics, and a dedicated low-distraction /glasses route.

How we built it

Lyrony is a TypeScript/React Web App built with Next.js-compatible Vinext and deployed as a Cloudflare Worker through OpenAI Sites. Browser APIs handle microphone capture, MediaRecorder encoding, and an AudioContext analyser for the live meter.

The server pipeline has two recognition paths. The pilot path sends the captured WebM audio to OpenAI's transcription API, retrieves timed lyrics for three known candidates, scores rolling lyric windows against the heard text, and uses the winning window to estimate playback position. The broader fallback sends the same sample to AudD for recording recognition. LRCLIB supplies runtime lyric timing when available.

GPT-5.6 receives only the small current lyric window and returns strict structured JSON with singable Romanization, a concise English gloss, and a sustained final syllable. The client maps phrase duration to a neon progress fill and hold rail. A separate glasses route demonstrates the minimal wearable presentation while keeping microphone processing in the paired-phone/backend architecture required by current hardware constraints.

How Codex was used

Codex was the primary build partner throughout the project. It helped research the current Meta Web App constraints, turn the product concept into separable recognition/language/rendering contracts, implement the responsive experience, and keep a public deployment and GitHub repository synchronized.

The most valuable Codex work happened during debugging. When the first page always displayed fictional Japanese lyrics, Codex traced the fallback path and replaced it with honest errors. When the microphone's animated bars did not prove the app heard anything, Codex added a real AudioContext meter and silence guard. When AudD's free quota became a risk, Codex helped narrow the demo to three supported tracks and build a transcription-plus-lyric-alignment path using OpenAI instead of disguising the provider failure.

The team made the core product and design decisions: no song selection, an international-first catalog, the pilot tracks, the Sing/Learn/Feel modes, pronunciation over literal transliteration, display glasses as the target, and the Lyrony identity.

How GPT-5.6 is used

GPT-5.6 is a functional part of the language layer, not decorative copy. Japanese and Korean lyric windows are transformed into a schema containing romanized, meaning, and hold values while preserving the exact number and order of timed lines. The prompt emphasizes performed pronunciation and long vowels so the output is useful for singing rather than only linguistically literal.

That structured output directly drives what the listener sees in the three modes and which final syllable the interface asks them to sustain.

Challenges we ran into

Proving the microphone was actually hearing music

An animated waveform looked convincing but initially moved independently of the microphone. We replaced it with real RMS measurements from an AudioContext analyser, disabled speech-oriented echo/noise processing for music capture, and added a silence threshold before sending a sample.

Recognition and lyrics are separate problems

Correctly naming a song does not guarantee synchronized lyrics exist. The pipeline now tries normalized metadata queries, supports both synchronized and plain lyric records, and labels estimated timing honestly.

Provider quota during a judgeable demo

AudD can return an error inside an HTTP 200 response, which the first implementation misreported as “No song matched.” We surfaced the provider's real error and then added a narrow OpenAI-powered fallback for the three certified demo songs.

Display-glasses constraints

The current glasses Web App surface does not expose the microphone directly. Rather than pretending otherwise, Lyrony separates the paired-phone capture/backend layer from the low-distraction wearable renderer.

Copyright and licensing

Karaoke products require appropriate lyric rights. The repository does not bundle commercial recordings or full commercial lyrics. The prototype retrieves only the runtime window it needs, and the production roadmap explicitly includes licensed synchronized lyrics.

Accomplishments that we're proud of

• Turned a speculative wearable idea into a live, judge-testable prototype in one build week.
• Preserved the defining zero-search interaction instead of quietly adding a song picker.
• Built an extensible line model that separates script, pronunciation, meaning, timing, and sustained sounds.
• Made the demo honest: real microphone feedback, visible silence detection, explicit provider errors, and no fictional fallback masquerading as recognition.
• Created a credible path from a three-song demo to more languages and broader recognition coverage.
• Kept the design expressive and understandable without losing the low-distraction wearable goal.

What we learned

Music recognition, lyric alignment, and pronunciation conversion are three different systems with different failure modes. A good experience needs to expose confidence internally while keeping the listener's interaction extremely simple.

We also learned that Romanization is not enough by itself. Singable guidance needs performed pronunciation, duration, and sustained syllables. The interface's color movement is as important as the text.

Finally, honest constraints strengthen a prototype. Separating the phone/backend capture layer from the glasses renderer made the architecture more credible than claiming unsupported direct hardware access.

What's next for Lyrony

• Build the native paired-phone companion with Meta's Wearables Device Access Toolkit.
• Add continuous streaming alignment so captions recover throughout a full performance.
• Integrate an authorized catalog covering recordings and live covers.
• License synchronized lyrics and build a human-review workflow for pronunciation packs.
• Expand from Japanese, Korean, and English into Mandarin, Thai, Hindi, Arabic, and more.
• Validate latency, battery, legibility, and safety on physical display glasses.
• Add community “Sing Packs” that let artists or authorized curators publish reviewed language layers.

Built With

• codex
• openai