Iris | Devpost

Inspiration

People are starting to hand real work to web agents like shopping, booking, forms, or account admin. Site owners now face a question they have no way to answer: Does my site survive when the visitor is an agent? We went looking for the tool that answers it, and found that nobody actually tries.

So the gap wasn't "run an agent." The gap was attribution. Steel's AgentBench leaderboard rates agents on sites. We wanted the missing half, which is rating sites for agents. Once we framed it that way, the shape of the answer was to have a controlled experiment, rather than a swarm.

What it does

Point Iris at a URL with zero context. There's no need for a task list, login, or configuration. It starts by conducting deep research into the customer base. Opens review sites, Reddit, forums and help centres in cloud browser sessions and builds an evidence pack of real quotes. Maps the site. A read-only crawler turns the live site into 3–5 executable journeys.

Then it writes a test brief. One Claude call turns the evidence pack plus the site map into 8 personas: segment, device, country, identity, goal, and the customer quote that justifies testing them. Two personas are a matched pair, differing in exactly one variable.

Iris then runs the swarm. 8 Steel sessions at once. Each persona gets its own mobile device fingerprint, residential geo proxy, and persistent profile. A DOM agent or a vision agent runs the goal and narrates every step. Agents never submit payment.

Once everything is collected, the final report is shown. Every stall is classified into an 11-category taxonomy, attributed to a single variable, matched to the research quote that predicted it, and given a concrete fix, ranked by how many customers it affects, each linked to the replay frozen on the frame where the agent gave up.

How we built it

Backend: Python 3.12. Everything talks through one file of Pydantic contracts (crucible/schemas.py), which meant four people could build against four seams in parallel without blocking each other:

Frontend: React 19, Vite, Tailwind 4. The app folds a stream of typed items into state through a pure reducer. Today a Player replays an authored per-target script; a websocket can replace it without touching a single screen. That's also our stage insurance: if Steel or the venue wifi dies mid-demo, the show still runs.

Challenges we ran into

Attribution is harder than running agents. Our first instinct was to run lots of agents and see what happened. That produces noise, not findings. Rewriting the matrix builder so that every config is provably one variable from baseline. Rejecting configs that aren't was the change that turned a demo into a product.

Agents fail for boring reasons. A large share of early stalls were our own fault: sessions that never started, models that timed out, browsers that disconnected mid-run. If you let those into the score, the score is meaningless. Building the harness-error bucket, retry-once, step caps, an inactivity release and a payment guard was unglamorous and took longer than the interesting parts.

Accomplishments that we're proud of

The controlled experiment actually holds. Every one provably one variable from baseline, enforced in code rather than in a code review.
Both engines completed live journeys on Steel. Browser Use baseline, Claude computer-use baseline, and a 3-config wave (baseline, mobile, returning) all completed end to end on real cloud browsers.
102 backend tests in about a second, with zero API keys and zero credits. Deterministic-first design made the system cheap to test.
A front end that survives a dead network. 24 unit tests across 17 files, plus a Playwright run that plays all five stages and screenshots each one.
We shipped the honest version. Harness errors are visibly excluded. Findings without a reproduction don't get reported. The score is smaller than it could have been, and it means something.

What we learned

Constraints are the product. "One variable from baseline" sounds like a limitation. It's the entire value proposition.
Be deterministic wherever you can afford to be. Pushing classification into plain rules and reserving the LLM for the genuine remainder made the system testable, fast, cheap, and much easier to trust.
Design the seams before the code. One shared schemas file let four people work in parallel for two days with almost no merge pain.
Measuring honestly is a feature. Excluding our own failures from the score, and refusing to report anything an agent didn't reproduce, is what makes the number worth looking at.
Infrastructure choices can be positioning. Mobile fingerprints, geo proxies and persistent profiles aren't implementation details here, instead they're the axes the whole product reasons about.

What's next for Iris

Close the loop. Take a proposed fix, apply it on a Steel Computer against a self-hosted store, re-run the matrix, and watch the score climb live. That turns Iris from a diagnosis into a treatment.
Finish zero-context discovery. The consumer-research and site-explorer stages and brief generation are built against their contracts and still landing; the FastAPI + websocket orchestrator will then replace the authored player without a single screen changing.
A public scoreboard. Steel's AgentBench leaderboard rates agents on sites. Iris rates sites for agents. Publishing readiness scores for the top e-commerce sites is the other half of that pairing.