Inspiration
I kept shipping agents that passed every eval I wrote. That only proves they handle the cases I already imagined. Prompt tweaking is the half everyone does. Finding the failures you never wrote a test for is the half that usually gets skipped.
What it does
Point Blindspot at an agent. It mutates inputs, runs the agent, groups new failure shapes, shrinks each finding, and writes a pytest file.
The main trick is a metamorphic check. Rename a vendor with a look-alike character. Reorder line items. Reformat money. The answer should stay the same. If it flips, that is a real bug, and you did not need a labeled correct answer to prove it.
Crash, schema, and metamorphic checks are plain code. An optional model lane only touches semantic mutation and a last-resort judge. With --fix, Blindspot also spawns one AO worker per failure class to patch the agent and open a PR.
How we built it
Python CLI (blindspot invoice / blindspot support). AO workers built most modules during the hackathon, and the tool can drive AO for the fix loop. Deterministic invoice runs make zero model calls. Optional providers are Groq and TensorMux.
Two toy agents carry planted blind spots so we could measure recall and before/after numbers for real.
Challenges we ran into
AO workers stall if you spawn them too fast, so dispatch spaces them. Coverage guidance looked smart and still did not beat random at this toy scale. I measured that and left it in the writeup. Hand-written metamorphic relations are powerful, and a bad relation will lie to you.
Accomplishments that we're proud of
Catches planted invoice blind spots 6/6 across seeds. Metamorphic gate stayed clean on the hand checks we wrote. Before/after metrics on the toys. AO is in the product, not a checkbox on the form.
What we learned
Eval sets flatter their authors. Failure discovery needs checks that do not need ground truth. An unmeasured claim is worthless.
What's next for Blindspot
Real agents instead of toys. Better relation packs. Tighter AO worker reliability. More honest judge budgets.
Log in or sign up for Devpost to join the conversation.