VibeTyping

Inspiration

We vibe-code everything now — describe intent, let the model do the work. But typing itself is still stuck in 1980. Meanwhile, cloud dictation ships your voice to someone else's server, and every "local" alternative feels like operating a tool, not talking to a teammate.

Then we noticed something: the Codex pet sitting on our desktop already was the perfect dictation interface. It's ambient, it's stateful, it's friendly. What if you could just... talk to it?

What it does

VibeTyping turns your Codex pet into a fully local voice keyboard:

• Hold a hotkey, speak, release — clean text appears in whatever app has focus. The pet animates through listening → transcribing → pasting states, so the system's state is always visible at a glance.
• Your phone becomes a remote mic: an Android app + voice IME streams audio over LAN to your Mac — audio never touches the cloud.
• Open models, all on-device: Whisper features drive turn detection that knows when you've finished a sentence, local ASR produces the final transcript, and gpt-oss is wired in as the local editor for spoken commands like "scratch that."

How we built it

The system is three pieces speaking one protocol:

1. Mac ASR server (Swift + MLX). A standalone Swift server runs Qwen3-ASR 0.6B (4-bit, MLX/Metal) for final transcripts, Silero VAD (CoreML) for silence candidates, and Smart Turn v3.2 (ONNX) for end-of-turn judgment — Smart Turn consumes Whisper log-mel encoder features we compute with MLX FFT. Everything is preloaded and pre-warmed at startup.

2. Endpointing cascade. Manual push-to-talk always works, but in auto mode: Silero proposes a silence candidate → Smart Turn judges whether the utterance is semantically complete → complete = auto-commit, incomplete = keep listening, with a 3-second silence fallback. Post-warmup Smart Turn inference averages 32.8 ms.

3. Clients on one wire. Both the macOS menu-bar client and the Android app/IME speak the same /v1/realtime WebSocket event protocol, so any client gets discovery (Bonjour), auth, and telemetry for free. The Mac client embeds a Codex-compatible pet runtime we built from the open-source Codex CLI pet format — it parses pet.json, slices sprite atlases, and maps dictation states onto standard pet animations (supporting both v1 8×9 and v2 8×11 atlas layouts).

We benchmarked against FLEURS (60 utterances):

$$\text{CER}{zh} = 10.84\% \qquad \text{WER}{en} = 4.46\% \qquad \text{RTF speedup} = 39.5\times$$

Challenges we ran into

• "Is it done talking?" is harder than "is it quiet?" Silence-based VAD commits mid-sentence constantly. Our Silero → Smart Turn cascade fixed complete sentences and hard cut-offs (100% on our turn benchmark), but TTS-generated half sentences with natural falling intonation still fool Smart Turn — a genuinely open problem we document honestly in our benchmarks.
• Android emulator audio is a haunted house. The emulator needs -allow-host-audio, macOS must grant mic access to the emulator's host process, and misconfiguration silently yields all-zero PCM. We built a structured E2E test harness (bundled FLEURS audio → real protocol → JSON evidence with per-stage errors and character accuracy) so "it works" became a measurable claim.
• Codex pet formats aren't one format. The desktop app's v2 contract (8×11 atlas, 16 gaze directions) differs from what the open-source CLI loader validates (8×9). Our runtime accepts both from day one.
• No streaming partials. Qwen3-ASR is whole-utterance; we compensated with fast inference and honest UI states instead of fake incremental text.

What we learned

• Endpointing is a semantic problem, not an acoustic one. The biggest UX gain came from Smart Turn, not from ASR accuracy.
• Evidence beats vibes (ironically). Structured E2E reports with request IDs, CER thresholds, and per-stage failures caught bugs that "looks fine in the demo" never would.
• The pet is not a gimmick. Making system state ambient and glanceable changed how the tool feels — nobody reads a status menu, everyone sees the pet.

What's next

• Finish the gpt-oss semantic-edit loop: spoken commands like "make that a bullet list" applied to the last utterance.
• Real-device Android validation over LAN, then public tunneling.
• CoreML/ANE backends so ASR stops competing with your local LLM for GPU.

Built With

• codex