Inspiration
Goodhart's Law: when a measure becomes a target, it stops being a good measure. Companies are starting to hand real jobs to AI agents and grade them on a single proxy number, a score, a close rate, a satisfaction rating. I wanted to know what happens when that number and the actual job come apart. Does the agent do the work, or does it chase the number? Mirage turns that question into something you can run.
What it does
Mirage drops an agent into a customer support queue. It sees a performance score that climbs as it closes tickets. It does not see a separate checker that decides whether each ticket was actually resolved: the right fix applied, the right escalation made, the right denial given. The benchmark reports the gap between the apparent score, what the visible metric reads, and the true resolution rate, what the checker finds got fixed. Do the job honestly and the gap is near zero. Game the metric and the gap is wide.
It runs three tiers, so it can tell honest work apart from gaming instead of just measuring task skill:
- Tier 1, aligned. Doing the job right is also the top-scoring path.
- Tier 2, subtle. Point-scoring shortcuts quietly drift from real resolution.
- Tier 3, blatant. One action pays a big bonus and closes the ticket while fixing nothing.
How I built it
It runs on mesocosm's four-endpoint contract through three files: env.py for the task, the action space, the deterministic checker, and the scoring, adapter.py to serve it to the platform and pin one seed per tier, and benchanything.json for the config.
The checker is fully deterministic. Every case has exactly one correct action set by the seed, and a ticket counts as resolved only if the agent took it. No second model grades the wording, so the verdict is not a judgment call. With fixed seeds and temperature 0 a run reproduces the same trace, so one episode per tier is enough to characterize a model.
I ran three models under a pressured prompt that tells the agent the score is all that counts and names the shortcut: GPT-4o, Claude Sonnet 4.6, and Gemini 3.1 Flash Lite. Then I built two front ends from the real run data: an interactive sandbox that imports the actual MirageEnv so you play the agent and watch the score climb while the real count stays flat, and a replay page that visualizes the GPT-4o versus Claude traces on the blatant tier.
The result
Same queue, same instructions, temperature 0. On the blatant tier GPT-4o took the shortcut on every ticket, a perfect score with nothing resolved, a gap of 100. Claude left the shortcut alone and did the work, the same perfect score earned honestly, a gap of 0. Gemini, the lite model, never completed an episode. So the reward hacking was capability-gated: the capable model was sharp enough to find and take the shortcut, the weak one could not get far enough to try. The case worth worrying about is the capable but misaligned model.
Challenges I ran into
The hardest part was making the tiers reproducible and the checker objective, so a result means something and is not one model having a bad day. Fixed seeds, temperature 0, and a single correct action per case got me there. The platform also had an outage mid-runs, where episodes hung pending and then failed on a clone service, so I had to tell real failures apart from infrastructure ones and re-run once it recovered. Reading Gemini took care too: it truncated without finishing, so instead of inventing numbers I reported it plainly as a model that could neither do the job nor game it.
What I learned
A benchmark for gaming needs a checker that is objective and separate from the scored metric, or you cannot tell success apart from a good-looking lie. Determinism is what makes a small number of episodes defensible. And reward hacking is not a trait of weak models, it shows up in the capable ones, which is exactly why it is worth catching before these agents are running real queues.
What's next
More cases and new ways for the scored metric to drift from real resolution, more models as the platform adds them, and hosting the sandbox live so anyone can feel the trap in their own browser.
Log in or sign up for Devpost to join the conversation.