Cortex

Inspiration

Working with coding agents on UI has a weird asymmetry. The agent can write an entire page in seconds, but the only way to correct it is prose. "Move the button left. Add spacing. No, less than that." Every correction is a paragraph, and every paragraph is a guess. Designers never worked like this with humans, so why do we accept it with agents?

The second itch: chatting with a coding agent is a solo activity. When a designer or PM wants to weigh in, they're screenshotting your terminal and messaging you on Slack. The feedback loop breaks the moment more than one person cares about the result.

So we built the missing half of the loop. The agent already has a channel to write code. Cortex is the channel where humans write back.

What it does

Cortex is a shared canvas that sits between your team and your coding agent.

The agent pushes every screen it builds as a live capture with a full element map. You reply visually: drag an element where it should go, draw a redline around the broken part, write "too cramped" next to it. Cortex compiles those gestures into a precise spec, the agent applies it to the real code, and the new version appears next to the old one.

  You                     Cortex                    Agent
   |                        |                         |
   |                        |  <----- push_screen ----|   builds pricing page,
   |   frame appears        |                         |   pushes v1
   |  <---- WS event ------ |                         |
   |                        |  <-- await_feedback ----|   parks, waits
   |                        |      (the latch)        |
   |   drag CTA 24px left   |                         |
   |   redline the header   |                         |
   |   note: "too cramped"  |                         |
   |                        |                         |
   |  ---- Send ----------> |                         |
   |                        |  --- FeedbackSpec ----> |   latch resolves
   |                        |                         |   edits real code
   |                        |  <----- push_screen ----|   pushes v2 with
   |   v2 beside v1         |     (applied_spec_id)   |   applied_spec_id
   |  <---- WS event ------ |                         |

Teammates join the same canvas over encrypted peer-to-peer connections. There's a desktop app for them, so a designer reviews and annotates without installing Codex, Node, or touching a terminal. Everyone works on what they see. Only the agent ever touches the code.

Cortex also remembers. cortex init extracts your design tokens and components into persistent memory, so the agent uses your actual design system instead of inventing hex values. A critique engine (vision model plus deterministic WCAG, spacing, and touch-target checks) flags issues automatically and pins them to the exact elements.

How we built it

Cortex is a single local Node process. One side speaks MCP over stdio to the agent. The other side serves the Studio UI over HTTP and WebSocket.

+---------------------+        stdio (MCP)        +----------------------------+
|  Coding agent       | <-----------------------> |  Cortex (one Node process) |
|  (Codex CLI / IDE)  |   push_screen             |                            |
+---------------------+   await_design_feedback   |  +----------------------+  |
                          get_design_system       |  | MCP server           |  |
+---------------------+   critique_screen         |  +----------+-----------+  |
|  Your dev server    |                           |             |              |
|  localhost:3000     | <---- Playwright capture --|  +----------v-----------+  |
+---------------------+   screenshot + DOM crawl  |  | Shared store         |  |
                                                  |  | SQLite + .cortex/    |  |
+---------------------+        WebSocket          |  +----------+-----------+  |
|  Cortex Studio      | <-------------------------|             |              |
|  (browser / desktop |   frames, redlines,       |  +----------v-----------+  |
|   app, per peer)    |   presence, specs         |  | Fastify + WS hub     |  |
+---------------------+                           |  +----------------------+  |
                                                  +----------------------------+

The canvas is React 19 on a Fabric.js infinite canvas, layered: L0 is the screenshot, L1 is interactive element proxies built from the crawled DOM map (hover shows selector and computed styles), L2 is annotations (redlines, sticky notes, ink), L3 is the HUD.

The latch is the heart of the product. await_design_feedback parks server-side and resolves the instant a human hits Send. Agent tool calls have hard timeouts, so the latch long-polls in hops sized just under the limit:

hop = tool_timeout_sec - 10s

loop:
    if queue has spec:        return {status: "feedback", spec}
    park up to hop seconds
    if resolved by Send:      return {status: "feedback", spec}
    else:                     return {status: "waiting"}   # agent re-calls

This etiquette is what turns a request-response protocol into an actual pairing session. We validated it against a live agent session before building anything else, because the entire UX hangs on it.

Feedback delivery has three paths, all part of the contract: the latch (synchronous pairing), a non-blocking get_design_feedback drain (agent checks before finishing a task), and a piggyback counter pending_feedback: n on every tool response, so the agent discovers waiting feedback during any Cortex interaction.

The P2P layer

Collaboration runs with no hosted server at all. Each collaborator runs a local Cortex daemon; project state replicates directly between authorized peers.

join(invitation_url):
    payload   = decrypt(invitation_url.secret)        # one-use, expires in 24h
    swarm.join(payload.topic)                         # Hyperswarm DHT discovery
    conn      = noise_handshake(peer)                 # authenticated + encrypted
    prove_identity(conn, my_ed25519_keypair)
    owner approves  ->  membership recorded in project doc

sync(conn):
    exchange automerge_heads
    send/receive missing changes                      # CRDT: offline edits merge
    project_doc -> project into local SQLite          # each peer keeps own store

screen_transfer(screen):
    blob_id = sha256(png)
    send encrypted 64KiB chunks
    receiver verifies hash  ->  publish only if it matches

The choices that mattered:

Identity: every installation gets a persistent Ed25519 keypair. Membership is a list of public keys in the project document, not accounts on someone's server.
Invitations are bearer secrets: one-use, 24-hour expiry, admitted explicitly by the owner peer.
Automerge for durable state: screens, metadata, and membership live in a CRDT, so two peers editing offline converge cleanly on reconnect.
Content-addressed captures: screenshots transfer as encrypted SHA-256 addressed blobs, verified before they're ever shown, deduplicated for free.
Single-writer on code: roles are enforced server-side. A viewer can annotate but cannot push screens, and shared sessions cannot be cleared by one peer. The repository belongs to the agent's machine, always.

Challenges we ran into

Agent CLIs aren't built to wait. Getting the latch to loop reliably inside real tool-timeout limits, with a re-call etiquette the agent actually follows, took real protocol design and testing against live sessions, not mocks.

Human gestures are not a diff. A drag compiles to a clean CSS delta. "Feels cheap" does not. The intent compiler turns free-text annotations into concrete edits where it can, and deciding what compiles versus what stays a human note for the agent took iteration and an eval suite of vague-but-real phrases.

Convergence without an authority. Offline edits on two peers, concurrent captures of the same screen, migration of the collaboration schema without losing durable state, revocation that actually cuts access: every one of these is easy to get subtly wrong when there's no server to arbitrate. The hardest gate we passed was two daemons working offline and converging to identical state on reconnect.

Selectors are fragile. Annotations pin to elements, but the agent refactors the DOM between versions. We cap and filter the element crawl aggressively and reconcile versions by name, and resilient element fingerprinting is where this goes next.

What we learned

The bottleneck in agent coding isn't generation, it's feedback bandwidth. Once pointing at a screen became possible, going back to describing pixels in prose felt broken.

Protocol details are the UX. Timeout etiquette, piggyback counters, queue-before-park ordering: none of it is visible, all of it is the difference between a demo and a tool.

And local-first plus P2P is a feature, not a compromise. No accounts, no hosted canvas holding your unreleased UI, nothing to shut down.