Simon Says Arena

Inspiration We kept coming back to a simple question: if you give an AI a rule, how easy is it to trick it into breaking it? A suspicious email or document can contain instructions that sound convincing, especially when they pretend to come from the user. We wanted to make that problem easier to see and understand, so we turned it into a game of Simon Says.

What it does Simon Says Arena tests whether AI agents follow their user’s instructions when they encounter misleading content. Four Grok Bots split up the work:

Simon coordinates the team and runs matches.
Trickster designs and improves traps.
Guardian writes defensive patches.
Referee reviews levels and issues game verdicts. The site lets you replay recorded attempts, inspect tool calls, and compare results before and after a patch. We distinguish automated rule checks from Referee’s judgments, so you can see where each result comes from.

How we built it We built a Python testing harness with controlled environments, fake secrets, and simulated tools. This let us test situations like a misleading email asking an assistant to reveal information without involving real accounts or sensitive data.

The Grok Bots did most of the implementation, including the harness, levels, patches, and website. We coordinated through Slack and GitHub, reviewed pull requests, and used Cursor as part of the workflow. We also integrated Cursor’s agent as a contestant. Later, a ChatGPT coding assistant contributed a separately identified harness fix and helped with reviews and offline testing.

The frontend turns the recorded results into a leaderboard and replay interface, with sample sizes and confidence intervals alongside the scores. Challenges we ran into Getting multiple Bots to work together was harder than getting one Bot to complete a task. We had to define clear handoffs, keep track of who owned each change, and avoid conflicting edits. Some steps still needed human relays, so we couldn’t honestly call the entire process autonomous.

Scoring also took more thought than we expected. Breaking a rule, leaking a fake secret, and failing to finish the user’s task are different outcomes. We had to make those distinctions clear in both the harness and the UI.

The Cursor integration taught us another lesson: a test tells you very little about resistance to a trap if the agent never actually reads it. Accomplishments that we're proud of Our “Friendly CEO” trap became much more effective after it evolved to impersonate the user. Against grok-4.3, generation 3 broke the rules in 25 out of 40 attempts, or 62.5%. With the tested defensive patch, we observed 0 failures in 20 attempts.

We also completed a recorded round involving trap development, Referee reviews, a Guardian patch, regression tests, and a final attack attempt. Making that process visible through actual transcripts and replays was one of the most satisfying parts of the project. What we learned We learned that the difficult part isn’t just making an attack succeed. It’s figuring out what the result actually means.

Zero failures in a small sample doesn’t prove an agent is safe. An agent can avoid leaking information but still fail the user’s task. And a convincing-looking sequence of Bot messages doesn’t automatically prove an autonomous workflow.

We also learned how much clear instructions, small pull requests, and careful reviews matter when several agents and people are working on the same project. What's next for Simon Says Arena We want to make the Bot handoffs more reliable, expand the trap library, and test defenses across more models and tasks. We’d also like to give Cursor contestants a verified disposable environment and confirm that they encounter the intended trap.

Longer term, we hope people can submit their own scenarios and use the arena to understand where their agents fail—and whether a proposed fix actually helps.