Inspiration
Face-to-face translation often begins with friction: choosing both languages, deciding who uses which side of the screen, and passing a phone back and forth. We wanted the interaction to feel closer to a shared interpreter placed between two people.
Mirror Talk turns one smartphone into a two-person table interpreter. Instead of assigning permanent “you” and “partner” roles, it organizes the conversation around the two languages it hears.
What it does
Setup begins with two short spoken sentences. The first confirms Language 1 and the second confirms a distinct Language 2. Either person can then hold the same central microphone, speak, and release it to translate.
For every turn, Mirror Talk records real microphone input, transcribes it, detects which language in the confirmed pair was spoken, chooses the opposite translation direction, displays the original and translated text across two language-specific histories, and optionally reads the translation aloud using AI-generated speech.
The interface defaults to English and includes a persistent English/Japanese switch that is independent of the spoken languages. Clear localized states cover microphone denial, silence, ambiguous language, an unexpected third language, unsupported audio, network timeouts, rate limits, provider failures, and blocked audio playback.
How we built it
The mobile interface uses Next.js, React, and TypeScript for a 390 × 844 viewport. Browser audio capture uses MediaRecorder and Web Audio APIs. A compact bridge between coral and indigo language surfaces contains one shared hold-to-talk microphone, playback control, live status, and the interface-language switch.
All OpenAI requests run through same-origin server routes, so the API key never enters the browser bundle. The pipeline uses gpt-4o-mini-transcribe for speech-to-text, GPT-5.6 (gpt-5.6-luna) through the Responses API and Structured Outputs for language detection, translation routing, and translation, and gpt-4o-mini-tts for optional MP3 playback.
The browser sends only the recording and the two confirmed language codes. It does not claim which person spoke or which translation direction to use. GPT-5.6 identifies the source language from the transcript, while the server verifies that source and target match the established pair. Ambiguous speech and languages outside the pair are rejected instead of being silently forced into a direction.
Responses API requests use store: false. Recordings are forwarded in memory and are not saved by the app. Conversation history exists only in the current browser session.
How we used Codex and GPT-5.6
The majority of Mirror Talk was built in a Codex session using GPT-5.6. Codex inspected and evolved the original static prototype, checked current OpenAI documentation, replaced demo timers with real server-side audio routes, implemented arbitrary two-language routing, preserved the selected mobile design, added automated tests, and packaged the app for deployment.
Codex also helped diagnose an iPhone Safari playback failure. The fix changed TTS output to MP3, reused and primed one persistent audio element during the microphone gesture, and added a visible Play fallback when the browser still blocks autoplay.
GPT-5.6 is also part of the runtime product. It receives the transcription as quoted speech, detects its language without assuming English or Japanese, selects the opposite member of the confirmed pair, and returns a natural translation through a strict JSON schema.
Challenges we ran into
The central interaction had to stay simple without making an unsupported claim that AI could identify a person by voice. We route turns using detected source language rather than speaker identity or separate microphones.
Reliable directionality for arbitrary language pairs required server validation of both language codes, rejection of duplicate and same-base pairs, verification of the model’s source and target, and explicit handling of third-language input.
Mobile Safari can also lose user-activation permission while the app waits for translation and TTS. Reusing an audio element and providing a fresh tap-to-play action keeps the translated text useful even when autoplay is blocked.
Accomplishments that we are proud of
- A working microphone-to-transcription-to-translation-to-history flow using real OpenAI APIs
- Automatic bidirectional routing with no fixed Japanese or English conversation side
- One shared microphone instead of separate controls for each participant
- Translation text that remains available even if TTS fails
- Server-only credentials, same-origin APIs, upload validation, rate limiting, and stable error responses
- English and Japanese interface copy without resetting the conversation
- Automated coverage for language-pair validation, direction routing, error states, and speech-playback races
- A verified non-Japanese Spanish/French flow in both directions
Potential impact
Mirror Talk could reduce setup friction in short, in-person multilingual exchanges while traveling, welcoming visitors, or helping someone at a reception counter. Its core interaction requires one phone, one microphone, and no accounts or shared-room setup.
The broader idea is that translation interfaces do not need to begin with fixed identities. For a two-language conversation, the language itself can provide enough routing information to keep the interaction lightweight.
What we learned
Separating interface language, detected spoken language, translation direction, and participant identity made the product more flexible and more honest. Structured Outputs let the server validate the model’s routing decision before presenting a completed turn.
We also learned to treat translated text and speech playback as separate outcomes. Committing text first makes the core translation useful even when mobile autoplay or the TTS request fails.
What is next
The MVP processes one completed utterance at a time. A future version could use the OpenAI Realtime API for partial captions during speech, followed by broader physical-device testing across languages, room acoustics, echo conditions, and mobile browsers.
Built With
- cloudflare-workers
- gpt-4o-mini-transcribe
- gpt-4o-mini-tts
- gpt-5.6
- mediarecorder-api
- next.js
- openai-codex
- openai-javascript-sdk
- openai-responses-api
- openai-sites
- openai-structured-outputs
- react
- typescript
- web-audio-api