Retrace

• 

  retrace

Inspiration

Rebuilding or migrating an app usually starts with someone clicking through the old one and writing notes about what each screen does and what it talks to. That knowledge lives in people's heads and goes stale fast. We wanted to record the app being used normally and turn that recording into something a team can read, correct and build from.

What it does

Retrace is a native macOS menu bar app. You pick one running app, press Start Observing App Activity, and work as you normally would. There's no task description to fill in.

While that app is in front, Retrace records:

• Clicks and scrolls, kept only when the element under the pointer belongs to the chosen app
• Named keys and shortcuts like return, arrows and cmd+s. Typed text is logged as a redacted burst and the characters are never stored
• Accessibility metadata (role, title, label, window title). Field contents are never read, and secure fields keep only their role
• Screenshots of only that app's windows, taken before, just after and once the screen settles after each action

Capture pauses on its own when you switch apps, lock the screen or open the Retrace menu.

When you press Stop & Review, a Sessions window shows every frame and event. You can omit anything and strip window or element titles, then confirm the review. Nothing leaves your Mac until you press Upload & Analyze, and the upload contains only what you kept.

The local workspace then analyzes the reviewed evidence in four bounded stages and produces a UI flow (screens and the actions between them) plus the API calls each action triggered. The dashboard groups every recording by app, one card per app, and each card gives you three things:

• API docs. Recorded traffic is collapsed into a short reference: endpoints grouped by host, then by method and normalized path (/api/invoices/{id}/send), with call counts, status codes, query parameters, redacted example URLs, first and last sighting, and the UI actions that triggered each one.
• Export for FigJam. The graph is validated against the session's evidence and then downloaded as JSON.
• Board link. Paste the FigJam board you imported into, and the card opens it from then on.

The FigJam plugin draws the flow inside one section as editable shapes and connectors. Steps are numbered and run left to right in the order you used the app, long flows wrap into a snake, and API calls sit directly below the action that caused them. Every shape keeps its evidence IDs, and labels that a person corrected are marked ✎.

How we built it

• macOS recorder: Swift 6.2, SwiftUI MenuBarExtra, ScreenCaptureKit for per-app window capture, and the Accessibility API with global event monitors for input. A separate RetraceCore module holds the manifest format, validation, review/redaction, local storage, Keychain credentials and multipart upload, so it can be tested without any permissions.
• Backend: FastAPI and Pydantic for loopback-only device connection, evidence ingestion and the four-stage analysis, running against NVIDIA Nemotron.
• Shared state: SpacetimeDB for private per-viewer state, incremental analysis jobs and preserved human corrections.
• Web: Next.js landing page and a local workspace with authorized SpacetimeDB subscriptions. Grouping recordings by app and turning traffic into API docs are pure functions with their own tests.
• FigJam plugin: validates the exported graph JSON and builds editable FigJam objects. The layout (BFS over observed screen/action edges, snake wrapping, API rows under their actions) is a pure module that runs under node --test with no Figma API.
• Optional: a mitmproxy helper for local network capture with sanitized export.

Challenges we ran into

• Scoping capture to one app. Global event monitors see everything, so each click is checked against the Accessibility element (or top window) under the pointer and dropped if it belongs to a different process.
• Recording keys without recording text. We classify key codes into an allow-list of named keys and shortcuts without ever reading characters, so typing shows up as a redacted burst.
• Timestamps. Evidence uses the monotonic uptime clock, the same base as NSEvent.timestamp, so a wall-clock change can't reorder a session.
• macOS permissions. Ad hoc signed builds look like a new app to macOS after each rebuild, so Accessibility, Screen Recording and Keychain access have to be granted again every time.
• Model limits. Our Nemotron endpoint is text-only, so analyzing the screenshots themselves still needs a vision-capable endpoint.
• Making the graph readable. Our first dashboard drew the graph on a free-form canvas, and it was hard to follow. We dropped it. The flow now lives in FigJam, laid out in interaction order where a team can annotate it, and the dashboard shows what people actually wanted to look up: which endpoints the app calls.

Accomplishments that we're proud of

• A privacy model that holds up when you read the code: review is the redaction step, sessions are stored with 0700/0600 permissions and excluded from backups, frames are re-hashed before upload, and the device token is only ever sent to loopback addresses.
• A validator (retrace-cli validate) that checks every session manifest: UUIDs, ordering, frame references, the key allow-list and secure-field stripping.
• Recording a real session against macOS Mail, with app-only screenshots and capture pausing correctly when Mail lost focus.
• API docs generated from what the app actually did, with IDs collapsed into path templates, secrets redacted from query strings, and each endpoint linked back to the click that triggered it.

What we learned

The privacy promises that matter most were the ones we enforced in code, like never reading AXValue and never storing characters. Putting those checks in a pure, permission-free core module meant we could test them properly instead of trusting manual runs.

What's next for Retrace

• Connect a vision-capable Nemotron endpoint so screenshots feed into the analysis
• Build the network helper into the recorder, so every session captures traffic without separate setup
• Developer ID signing and notarization, so permission grants survive updates
• Verify FigJam import end to end on a live board

Built With

• fastapi
• figma-plugin-api
• macos-accessibility
• next.js
• nvidia-nemotron
• pydantic
• python
• react
• screencapturekit
• spacetimedb
• swift
• swiftui
• typescript