Relay

Relay lets Codex generate, test, and install its own MCP adapters, giving it the power to operate any application on your computer.

Comment

Inspiration

Codex is able to reason about almost any application on your Mac, but it is not able to directly interact with most of them. For example, games, native macOS apps, and old internal tools are often not considered by an agent unless someone manually builds the integration out for each one. This results in a slow repetitive process that means most applications won't get supported.

What it does

Relay turns Mac applications into tools that Codex can call through the MCP. It consists of 2 parts:

  1. An adapter generation which allows users to provide an intent such as "Control Spotify with tools like play, pause, and skip". Relay can use codex to design the toolset, write the local MCP adapter, run a smoke test, and register it. If the adapter fails the test, it never gets exposed to the agent.
  2. Seven reference adapters. We built adapters for Minecraft, Clash Royale, Chrome, Messages, OBS, AppleScript, and an iMessage listener. Together, they show that the same interface can work across very different kinds of applications, including network protocols, browser injection, AppleScript, SQLite, synthetic mouse input, and app-specific scripting APIs.

How we used Codex and GPT-5.6

Codex is not just one feature inside Relay. It is the runtime behind the entire system.

Relay launches codex app-server as a child process and communicates with it over newline-delimited JSON-RPC through standard input and output.

The adapter generator uses two Codex turns on the same persistent thread.

During the first turn, Codex has read-only access. It reads our adapter contract and the application documentation, then proposes a toolkit as a structured JSON specification.

During the second turn, Codex receives write access limited to the adapters/ directory, with network access disabled. It uses the proposed toolkit to write the actual server.mjs implementation. Since both turns share the same thread, Codex still has the reasoning and context behind the original tool design.

At runtime, Codex also decides how to use the generated adapters. Our backend does not manually invoke individual tools. It starts a Codex turn and streams the resulting events while GPT-5.6 chooses which tools to call and in what order.

The workspace UI retrieves the available models directly from the app-server through model/list. Users can choose between models in the GPT-5.6 family, with GPT-5.6-Sol selected by default.

We also used Codex heavily while building Relay itself. One of our adapters, chrome-mcp, even completed our hackathon submission form. It filled everything out correctly, but refused to click Submit because that restriction was enforced inside the adapter rather than left to a prompt.

How we built it

Relay has a Fastify backend that manages one long-running codex app-server process.

Multiple chat threads share that process through stdio JSON-RPC. Because overlapping turns can interfere with one another, requests are serialized using a promise-chain mutex, allowing concurrent requests to queue safely.

Turn events are streamed to a Next.js and shadcn workspace interface using Server-Sent Events.

Each MCP adapter runs as its own child process over stdio. New adapters can be loaded without restarting the main app-server.

Challenges we ran into

Working with real desktop applications meant dealing with a lot of undocumented and inconsistent behavior.

Messages AppleScript support is becoming unreliable. Several documented properties now fail with error -1728. We had to work around the missing functionality and make the adapter report whether a message was actually confirmed instead of assuming the send succeeded. Sending a message behaved differently depending on the conversation. Sending into an existing thread could block indefinitely, while creating a new conversation required waiting for the full round trip. We ended up using separate send paths for each case. CSS selectors broke in unexpected ways. We initially used nth-of-type as though it meant “the Nth input on the page,” but it only counts matching sibling elements. This caused checkboxes to fail during a live run. We now generate full ancestor-based selectors and verify that they resolve back to the intended element. React fields can appear filled while still containing no submitted value. Directly assigning .value does not reliably update React’s internal state. We now call the native prototype setter, dispatch input and change events, and read the value back to confirm that the update worked. Headless Minecraft rendering used too much memory. Each first-person renderer consumed roughly 430 MB, and the cost multiplied when old adapters were replaced. We removed the renderer and capture the actual screen instead. This also made the demo easier to follow because the audience sees exactly what the agent sees. stdout has to remain completely clean. MCP uses stdout for protocol messages, so even one accidental log line can corrupt the transport. All diagnostics are written to stderr and a log file instead.

What we learned

The biggest lesson was that one shared contract can work across very different applications.

The adapter might communicate through a network protocol, an accessibility tree, injected JavaScript, AppleScript, or synthetic mouse clicks. Codex does not need to know the difference. It only needs a reliable set of tools.

We also learned that safety rules should live inside the adapter, not only inside the prompt. For example, chrome-mcp blocks submit-like controls unless the caller explicitly opts in. Even if the agent misunderstands the task, it can fill out a form and stop before taking the final action.

What's next for Relay

We want to improve the toolkit review process, generate adapters for more applications, and make artifact handoff between apps a first-class part of the workflow.

Built With

Share this project:

Updates