AppAtlas

Verified navigation
Grounded Analytics
Codex Plugin
Seamless integration

Inspiration

Point an AI agent at a piece of software it has never seen and watch what it does. It takes a screenshot. It guesses where to click. It takes another screenshot. It guesses again. Click by click, it feels its way through the interface like someone navigating a dark room, and every one of those steps is a model call that costs tokens and time.

We kept hitting the same wall. Agents are getting genuinely good at deciding what to do, and they are still terrible at knowing where things are in an interface they were not trained on. The knowledge exists, it is just locked in the UI, and nobody has handed the agent a map.

So we had a simple thought. Humans do not rediscover an app every time they open it. They build a mental model once and reuse it. Why should an agent start from zero on every single task? What if we crawled the interface once, turned it into a map, and let any agent read that map instead of guessing?

That is AppAtlas. Computer use, but with a map.

What it does

AppAtlas crawls a web application and builds a navigable graph of it, then serves that graph to any AI agent over MCP.

The crawl does not stop at pages. It captures every individual control: the dark mode toggle, the font size radios, the email field, each one as a first-class node carrying the exact selector needed to operate it. That distinction is the whole point. Most crawlers can tell an agent "go to the Appearance page." AppAtlas tells it "toggle Dark mode," and hands over the selector to do it.

Ask it a question in plain language and it retrieves over two signals at once, keyword and semantic, and uses whichever performs better. So "see my past bills" finds the Invoices page even though the two share no words, and the whole thing still holds up whether the embedding model runs on your own device or through an API.

The answer is not a guess from a language model. It is a real path through a real graph, with real selectors at every step.

And all of it is exposed over an MCP server. Open a fresh Codex session that has never seen the application, ask "how do I turn on dark mode in this app," and it calls AppAtlas on its own and comes back with the steps. Zero integration. One tool call instead of eight browsing steps. The agent never even needs access to the app itself, because it is reading the map, not the software.

There is also an analytics side. Because the interface is now a graph, AppAtlas can report on it: dead ends, orphan pages, the most connected screen, how many clicks it takes to reach any setting, and which questions users keep asking, which is really just a map of where the design confuses people.

How we built it

The whole project was built with Codex on GPT-5.6 Terra, across a series of structured sessions. Each session had a written spec and explicit acceptance criteria, and Codex did the implementation while we specified and measured.

The single decision that shaped everything: we wrote the evaluation harness before we wrote the search. Twenty-one natural language queries, each mapped to the node it should resolve to, including deliberate no-answer cases the system is supposed to reject. From that point on, every change to retrieval was measured against accuracy and recall instead of judged by feel. When we made controls first-class nodes, we could see exactly what it bought. When we added semantic retrieval, we could see whether it actually helped or just felt like it should.

The stack:

Crawler: Playwright, breadth-first, capturing pages, controls, selectors, and screenshots.
Graph: NetworkX, with pages and controls as nodes and navigation and containment as edges.
Retrieval: a hybrid of TF-IDF (word and character n-grams) and embeddings, blended and thresholded, degrading gracefully to lexical when no embedding endpoint is available.
Backend: FastAPI, provider-agnostic through any OpenAI-compatible endpoint, so the LLM can be OpenAI, a compatible hosted API, or a fully local model.
MCP server: four tools (answer_question, find_workflow, search_interface, analyze_interface), packaged as a Codex plugin.
Frontend: React and Vite, with a live graph of page thumbnails, path highlighting, and an analytics dashboard.

Challenges we ran into

Silent failures, everywhere. This was the theme of the whole build. An unloaded .env, a model name that did not exist, an embedding cache rejected on a dimension mismatch. Every one of them degraded quietly to a working-but-worse fallback instead of raising, so the system looked fine while being wrong. We lost real time to answers that were plausible and quietly lexical. The fix was to make every fallback loud: log which one fired and why. You cannot build a reliability tool on top of code that hides its own failures.

The graph modeled pages, not controls. Early on, a toggle had nowhere to live in the graph, so questions like "how do I enable dark mode" could only ever resolve to the page the toggle sat on. Making controls first-class nodes was the change that mattered most, and the eval harness proved it: it was the single biggest jump in retrieval accuracy we measured.

Semantic search that shared no words. "Night mode" and "Dark mode" have zero tokens in common, so pure keyword matching scored them at exactly zero. Getting embeddings to work meant discovering that the model needed specific asymmetric prefixes for queries versus documents, which nothing was sending. Until that was fixed, adding a semantic layer made retrieval worse, not better, and only the eval harness made that visible.

The demo agent has to reach for the tool on its own. An MCP server is invisible if the agent never calls it. Getting Codex to pick find_workflow unprompted came down to the tool descriptions and a skill file telling it when to use the map instead of guessing. If you have to name the tool, the descriptions have failed.

What we learned

Give agents knowledge, not integration. The cleanest way for an agent to operate software is not to wire into its internals. It is to read a map built from the outside. AppAtlas never touches the target app's code, which is exactly what makes it drop-in for any application.

Measure first, or you are guessing too. Writing the eval harness before the search was the best decision in the project. Every "this feels better" turned into a number, and more than once the number said the opposite of the feeling.

Reach for a map, not a guess. A path from a real graph with real selectors is a fundamentally different object from a path a language model invented. One can be verified. The other just sounds right, which is the exact failure mode we set out to remove.

What's next for AppAtlas

Map your entire internal stack. Every admin panel, dashboard, and internal tool a team relies on, mapped and navigable by any agent.
Documentation that cannot drift. Guides derived from the live interface, so they are never out of date with the product.
Change intelligence. Diff an interface across versions to catch breaking UX changes before users hit them.
Beyond anchor navigation. Support single-page apps and button-driven flows, so the crawler reaches interfaces that are not just linked pages.

As teams hand more real work to agents, the interfaces those agents touch need a map they can trust. AppAtlas is that layer.