Incidents | ChatGPT
Incidents
Releases
Cases

Inspiration

I am an AI researcher and I fight with evals of my agent systems every day. When I heard about WebMCP, I thought it was a perfect fit for data analysis. I usually have a concrete question, and the road from the question to finding the data and visualising it is long. Then a deeper question appears, and I wander down a tree of questions looking for the answer. WebMCP and analytics dashboards are, in my view, a beautiful pair.

What it does

Catchfly is observability for WebMCP apps, built so an agent can run the investigation next to me. The @catchfly/sdk captures production traces of tool calls, while the @catchfly/cli runs Chrome WebMCP Evals in CI and uploads the report. The dashboard joins both sources per release:

Incidents
Releases
Sessions
Regression Explorer
Cases

The dashboard itself

The dashboard itself is a WebMCP app with thirty-eight tools. I ask:

"What broke in the latest releases?" The agent reads the incident overview. I ask: "Open the evidence." The view switches on my screen, attributed, with an Undo button. I ask: "Show me the production sessions that failed on this tool." The filters set themselves.

I open one, the agent drafts an eval case from it, shows the draft and saves it after my approval. catchfly eval pull brings it into the repo. The tree of questions that used to take me an afternoon fits in one conversation.

All of this is written in the lovely Catchfly UI.

How we built it

Catchfly is built as a TypeScript monorepo:

React and Zustand for the dashboard
API as Netlify Functions
One route table shared with:
- a standalone server
- Vite middleware
PostgreSQL via Supabase

In the app, human clicks and agent tools call the same state actions. Every write returns the resulting state, every change has a source, the human can undo it, and the agent learns about the undo on its next call.

Demo environment

For the demo, I built a fictional Devpost Review Console with a real WebMCP surface so the Chrome runner has something to evaluate. I also created a deterministic synthetic world containing:

18 releases
25,000 sessions
27,000 eval attempts
a deliberate latency false lead

I also tested how the app worked in my previous projects using the Catchfly SDK, and everything worked perfectly.

Challenges we ran into

The biggest challenge for me was to design the UI/UX in such a way that the application looks good and is easy to read, both for users and in the tools that support WebMCP, like ChatGPT Browser.

At the same time, I’m racking my brains over WebMCP vs. security, but I haven’t come up with much yet.

Accomplishments that we're proud of

I’m really pleased that it’s an end-to-end product. I’m posting it on Discord the day before the hackathon ends, and everyone can still use it because it works and looks cool.

What we learned

The most important thing: I discovered and learnt WebMCP.

It’s great fun - but I wasn’t the only one to notice this, as basically everyone on Discord had a brilliant time collaborating with the agents via ChatGPT.