Hermes Live Companion

• 

• 

• 

• 

Inspiration

Most AI demos focus on one capability at a time: chat, voice, vision, or tools. I wanted to explore what happens when these capabilities are coordinated inside one observable, interruptible session.

Hermes Live Companion began as a practical experiment: could a local-first control surface combine natural voice interaction, visual input, persistent session context, tool approvals, interruption control, and runtime observability without hiding the system behind a simple chat box?

The project was built during OpenAI Build Week with Codex as the engineering collaborator and GPT-5.6 Sol as the reasoning model inside the Hermes runtime.

What it does

Hermes Live Companion is a Windows-first multimodal agent interface that allows a user to:

• speak naturally and receive streamed spoken responses;
• preserve conversational memory across multiple turns;
• interrupt the agent while it is thinking, using tools, synthesizing, or speaking;
• approve and capture a single camera frame for visual analysis;
• ask follow-up questions using the same visual and conversational context;
• inspect the active model, runtime state, tool activity, and latency events;
• reset or stop the active session at any time.

The interface keeps the user in control. Camera access is explicit and single-frame, protected actions preserve Hermes approvals, and runtime activity remains visible throughout the session.

How it was built

The browser interface uses React, TypeScript, Vite, and the Web Audio API.

A FastAPI backend manages the session state machine, WebSocket communication, audio processing, camera frames, interruption behavior, structured logs, and health monitoring.

The Companion launches a separate loopback-only Hermes worker through authenticated JSON-RPC. Hermes provides the active GPT-5.6 Sol session, tool access, approvals, and conversation memory.

The voice pipeline includes:

• Silero VAD for speech detection;
• faster-whisper with optional CUDA acceleration for local speech recognition;
• sentence-level streamed speech output;
• a persistent Kokoro TTS worker with Edge TTS fallback;
• acoustic, headphone, and push-to-interrupt operating modes.

For vision, the user or Hermes can request one approved frame. The frame is previewed, validated, attached to the active session, analyzed by GPT-5.6 Sol, and deleted after the turn.

How Codex was used

Codex worked as an engineering collaborator throughout the project.

It helped inspect the existing Hermes environment, design the architecture, implement the backend and frontend, create contract and smoke tests, benchmark speech pipelines, review interruption behavior, harden privacy boundaries, package the Windows release, and audit the final GitHub submission.

Repository-level rules constrained the work:

• no modification of the production Hermes installation;
• no reuse of desktop tokens or ChatGPT cookies;
• loopback-only networking;
• preservation of tool approvals;
• verification before each release milestone;
• no credentials, raw audio, captured images, sessions, logs, or local paths in the public package.

Challenges

The most difficult part was coordinating several asynchronous systems at once.

Voice activity detection, speech recognition, model streaming, tool calls, text-to-speech, playback, interruption, and camera approval all have different timing and failure modes. A useful interface also needed to show what was happening without overwhelming the user.

Another challenge was privacy. The project had to remain demonstrable while keeping credentials, raw media, local session data, and development history out of the release.

Portability was also difficult because CUDA, browser permissions, microphone behavior, camera access, Hermes authentication, and Windows audio devices vary between machines.

Accomplishments

The final Build Week version successfully demonstrated:

• a real GPT-5.6 Sol Hermes session;
• streamed multi-turn voice interaction;
• session memory and reset;
• approved camera capture and visual analysis;
• image-context follow-up questions;
• interruption and stop controls;
• visible runtime and latency activity;
• privacy-conscious logging and packaging.

The release passed 55 Python tests, 9 frontend tests, the TypeScript/Vite production build, Python compilation checks, PowerShell script validation, and a real GPT-5.6 end-to-end runtime test.

The public release history was rebuilt as a clean zero-parent commit so older private development paths and local environment details were not exposed.

What I learned

Model intelligence is only one layer of an agent product.

A usable agent also needs state management, permissions, interruption, observability, privacy boundaries, recovery paths, and clear human control.

I also learned that Codex can contribute far beyond code generation. With clear constraints and verification requirements, it can participate in architecture, testing, release engineering, security review, documentation, and deployment.

What's next

Future versions could add:

• lower-latency audio processing;
• continuous or event-driven vision;
• more local and cloud model routing options;
• cross-platform installation;
• additional approved tools and sensors;
• replayable sessions and richer runtime diagnostics.

Hermes Live Companion is an exploration of what a local-first, observable multimodal runtime can look like when voice, vision, memory, tools, and human control are designed as one system.

Built With

• agent-orchestration
• codex
• computer-vision
• cuda
• fastapi
• faster-whisper
• gpt-5.6-sol
• hermes-agent
• kokoro-tts
• local-ai
• multimodal-ai
• openai
• python
• react
• silero-vad
• speech-recognition
• typescript
• web-audio-api
• websockets