Bluff Lab

• 

  cover

• 

• 

• 

• 

• 

Inspiration

I really like social deduction games. They're also a clean model of problems agents have now: an agent that carries memory between encounters has to decide whom to believe, and that memory can be lied to, or poisoned. A bluffing card game allows you to verify who lied immediately, so every belief the agent forms can be checked against ground truth.

What it does

Bluff Lab is a controlled experiment in whether agent memory makes agents better or more exploitable. Three players play a simplified Liar's Deck. The agent, the Profiler, is a frozen Claude model that proposes hypotheses about each opponent between games; a numeric layer scores them against revealed bluffs.

• Mind Inspector: the recorded decision variables behind any decision.
• Memory Surgery: re-run the same decision, or the same seeded games, with memory erased, rolled back, minus one line, or with a planted false memory.
• The Long Con: an opponent that earns trust, then defects.
• Strategy Forge: Claude compiles its memory into a Python strategy, run as untrusted code in a network-blocked Daytona sandbox.
• Voice: ElevenLabs voices the agent's questions.

The problem: agent memory claims are usually untestable because ground truth is difficult to verify. Here every belief has a verdict.

How we built it

deterministic game engine with event log across daytona boxes. For our main agent (The Profiler), claude proposes hypotheses as predicates and then a numeric layer scores them against reveals. I used surrealDB for the typed temporal beliefs in agent memory, Arize over frozen decisions for observability, then elevenlabs for voice that adapts dynamically (like suspicious or nervous sounding agents).

Challenges we ran into

Making a good baseline is difficult--my first attempt trivialized the exchange and had a lower brier than profiler (trivializing baseline was classifier based on word count). I also had a reflection prompt accidentally leaking the planted tell.

Accomplishments that we're proud of

Memory made a frozen model play better, with no retraining. Memory you can operate on: On the same frozen decisions, scored as Arize experiments, erasing memory flipped 63/555 decisions. A planted false memory flipped 0 for our evidence checking agent and 146/55 for a provenance blind one. Strategy Forge: the agent compiled its memory into a Python strategy, which ran as code in a Daytona sandbox. Over 600 games it won 60.3% (CI 56.5 to 63.8), nearly double the Profiler's 34.7% (CI 31.5 to 38.0). A forge given no memory won 22.5%, so the gain came from what it had learned

What we learned

Against consistent opponents, Claude's hypotheses helped (+9.5 points; exploratory, since that control was added after pre-registration). Against a long con, the same hypotheses made the agent more exploitable: with reflection on, it let through 23.8 more of the betrayer's bluffs (CI +17.8 to +29.8) and noticed the turn later

Also another important thing is the structure was imperative (raw transcript didnt help compared to a hypothesis)

bayesian memory let through far fewer long con bluffs than no memory at all. The vulnerability came from the beliefs the LLM wrote.

What's next for Bluff Lab

teams with shared team memory. maybe even collaborative game theoretic alliances, or a compromised teammate would be cool.

Built With

• amazon-web-services
• arize
• claude
• daytona
• embeddings
• surrealdb