mortiphi

Inspiration

Codex as a feature had side-chats, and often, to understand the very verbose description of a plan or what was done, I would start a side chat. There would be a long back and forth - more about understanding how the agent thinks and communicating how I would think. Once done, finally, I would send a handoff prompt to the main chat. This was done to prevent context pollution in the main thread. However the difference was, for my purposes of understanding the project and communicating my thought, the mode of communication did not need to be text and I found myself preferring to speak instead. WisprFlow would understand the intent of what I was saying and that's great but now as agents get smarter than me, I need to understand what the agent wants to say, to convey my intent better.

What it does

Turn any OpenCode thread into a Voice agent with one command. Just command /mortic and you start speaking in an ephemeral fork of the main thread (only on OpenCode for now). This conversational interface populates a sidepod in the TUI itself and starts a popup in the GUI. The fork inherits the main thread's reasoning, uses a full duplex architecture with acoustic echo cancellation and eager end of turn, handles near-instant self-compaction using Mercury 2 (a diffusion model) while the user speaks, and also has a speak-screen bipartite scheme in the structured output so the conversation fits how a normal human conversation would be like.

How we built it

We built it for OpenCode using GPT 5.6 Sol, primarily to refine the harness and nature of speak-screen response, improve recall and tune compaction frequency. We defined an eval-harness with expectations on what we expect from the Mortic agent, what is unacceptable in terms out output as naturally as a coding agent fork normally outputs large amounts of markdown and code itself which is unacceptable for speech. This was a large scale open-ended agentic task where the model itself trained or fine tuned a better harness for our task.

Challenges we ran into

We had originally built it over the Codex App Server - but we felt that the latency was not great considering the nature of the tasks and the autoregressive nature of the model because reasoning itself would be quite slow, even though TTFT would be great, as chain-of-thought was still autoregressive and linear.

Within OpenCode, we had to suppress markdown, handle echo cancellation vs. eager EoT, and code vs speak handling. We also had to rebuild the entire agent and harness build using 5.6, which did a complete refactor based on SOTA approaches to handle EoT as well as AEC, as the version we had before that was an unreliable mess.

Accomplishments that we're proud of

Usage costs for Mercury 2 + STT + TTS, for our use case is much lower than GPT Realtime 2.1 API, offering nearly triple the number of hours, and we are now using Mortic for our own use cases. Despite the bottleneck with Codex app server, we pivoted entirely and managed to build an alternate product that works a lot better for us.

We can process complex prompts within 2 seconds, and have almost never run into speak-screen issues while normal usage.

What's next for mortiphi

We will expand the product for Codex, once Cerebras usage goes live and we can test the product with 1000 TPS inference. We will also close the handoff product, and start moving towards a voice native coding harness that allows for users to use it side by side.

Built With

• bun
• cartesia
• deepgram
• fastapi
• inception-labs
• javascript
• livekit
• macos
• mercury-2
• node.js
• opencode
• opentui
• portaudio
• python
• rest-api
• server-sent-events
• solidjs
• sqlite
• typescript
• uvicorn
• voice
• webrtc
• websockets
• zod