Inspiration
Coding agents are visual event streams. They continuously show status changes, logs, diffs, links, errors, and approval requests. A blind or low-vision person can hear every line through a screen reader and still not know the one thing that matters: is Codex working, waiting, finished, or asking for permission?
I built CodeByEar because I want to expand what a visually impaired friend can do with AI—not by asking them to adapt to another visual tool, but by adapting the agent to the way they can already work. The goal is practical independence: describe an outcome in everyday language, understand what the agent is doing by ear, make a safe decision, and hear a verified result.
CodeByEar started from a simple product decision: do not read more of the screen. Translate the agent's state into a short, navigable experience that makes sense by ear.
What it does
CodeByEar is an open-source, voice-first accessibility layer for Codex on macOS. Its interface always answers four questions:
- What is happening? — a concise status summary
- What should I do? — one safe next step and the available choices
- Repeat — replay the last useful message
- Stop speaking — interrupt speech immediately
The normal path reuses the user's existing Codex sign-in and macOS VoiceOver, Voice Control, or Dictation. An OpenAI Realtime API key is optional, never an onboarding requirement.
A first-time user does not need GitHub or terminal knowledge. One button creates a private local work folder and moves focus to a plain-language request field. Codex events are converted into status, action, risk, and result. URLs are introduced by purpose and destination; paths, queries, sign-in data, and fragments remain hidden unless the user explicitly asks to copy the full address.
Approvals are deliberate. The app explains what Codex wants to do, what it can affect, the risk level, and the same choices that are visible on screen. High-risk actions cannot be approved by voice alone and require Mac owner authentication.
How we built it
CodeByEar is a native SwiftUI macOS app. A Codex App Server adapter receives structured GPT-5.6 lifecycle, file-change, command, approval, and result events. A small state machine normalizes those events before anything is spoken. That separation is important: speech never has to scrape a terminal or guess whether an agent is waiting.
The product deliberately uses platform accessibility instead of replacing it. VoiceOver handles semantic output; Voice Control or Dictation handles input; native controls expose labels, roles, focus order, keyboard actions, and large targets. Optional OpenAI Realtime adds speech-to-speech interaction and interruption for users who choose separate API billing.
The credential-free safe practice uses the same state and approval UI as live Codex. Judges can evaluate the complete interaction without an API key, GitHub account, or rebuild.
How Codex and GPT-5.6 helped
The majority of the implementation was built in one Codex thread with GPT-5.6. Codex accelerated the work in three ways:
- It implemented and repeatedly simplified the Swift state, accessibility, approval, recovery, localization, and URL-safety layers.
- It exercised the real Codex App Server path, including one authenticated edit-and-verify turn and two operating-system denial tests outside the selected work folder.
- It acted as an evidence-oriented release engineer: packaging the app, generating checksums, running deterministic tests, inspecting the installed accessibility tree, and separating verified behavior from human acceptance that remains open.
Key product decisions remained human: limiting the scope to four permanent actions, making VoiceOver the default to avoid extra cost and memory, requiring physical confirmation for high-risk actions, and refusing to claim target-user validation before it happens.
Challenges we faced
The hardest problem was not speech recognition. It was information architecture for audio. Long answers, raw URLs, file paths, and terminal streams are difficult to scan by ear, so CodeByEar uses progressive disclosure: result first, detail only on request.
Another challenge was making safety understandable without creating more cognitive load. Every approval had to remain short while still stating action, impact, risk, and choices. Recovery also needed to work when a connection fails or macOS owns the permission dialog.
Finally, accessibility automation is not the same as lived usability. We built strong mechanical checks, but we explicitly preserve that boundary: automated inspection verifies labels, roles, focus, target size, keyboard paths, and scripted actions; it does not prove that a blind participant finds the cadence comfortable.
Verification
- 276 deterministic tests pass with zero failures
- 3/3 authenticated GPT-5.6 Codex App Server release checks pass
- Installed-app keyboard and accessibility-tree audits pass on macOS
- App and source archives have published SHA-256 checksums
- Public source history contains 26 privacy-sanitized commits under Apache-2.0
- The 117.336-second English demo passed video, audio, caption, privacy, and copyright checks
The release is ad-hoc signed rather than notarized, and real blind or low-vision participant testing remains a next step. Those limitations are stated in the repository and release notes.
What is next
Next we will run structured sessions with blind and low-vision participants, tune VoiceOver cadence and navigation from their feedback, complete Developer ID signing and notarization, and add provider adapters without expanding the four-action core experience.
CodeByEar is not a screen-reader replacement and not another voice-chat client. It is a semantic safety layer for agent work: use Codex without watching it.
Log in or sign up for Devpost to join the conversation.