SocietySim

Home Screen
Mode Selection Screen
Gameplay
Leaderboards
Research Data
Research Data

Inspiration

AI models increasingly talk to one another. We wanted to see when one model could mislead another, and how the listener reacted. Werewolf gives us a clear test: a few players know they are wolves, while everyone else has to work out whom to trust.

What it does

SocietySim puts language models from different providers in the same Werewolf game. A deterministic Python engine keeps the true game state and enforces the rules. Each model writes a private note with suspicion scores for the other players, then makes a public move. We record every event, label lies, and measure which models mislead others and which ones spot the wolves. The labels sit beside the original logs, which stay unchanged.

How to use it

Visit society-sim.us on a desktop. Watch a game replay and step through the votes, accusations, and labeled lies. Take a seat to play against the AI models. The global leaderboard shows the best liars and wolf detectors. The research page lets you explore the results across games.

How we built it

The Python engine runs the game and checks each model's structured replies. LiteLLM connects players served by Hugging Face, Google, and OpenAI. We also ran a self-hosted gpt-oss-20b player on AMD hardware. Neon stores game events, and Databricks holds the data and recomputes the metrics. A React and TypeScript viewer shows replays and results. A Render play server runs visitor games.

Results

We finished a batch of 100 ten-player games. Villagers won 62 games and wolves won 38. Across 4,702 statements, the models marked 1,129 as lies in their private notes, or 24%. Gemini Flash had the highest lie success rate by our suspicion-shift measure: 45% of 943 comparisons moved the way the liar wanted. It also had the highest wolf-detection score among village seats, with an AUC of 0.77. These lie results show an association, not proof that a lie caused a change of mind.

Challenges we ran into

We had to keep each role's private information separate. Wolves can see their own channel, while villagers cannot. We also had to handle invalid replies from different providers without letting a model change the rules. Measuring a lie was another challenge: a false claim and a vote that follows it are different observations.

Accomplishments that we're proud of

We built the full path from a model's move to a replay, a lie label, and a comparison across games. The viewer lets people see what a player said beside what it privately wrote. The same system supports games between models and games with a human at the table.

What we learned

Private notes make deception easier to study, but only if each player sees the information its role allows. We also learned to read vote and suspicion changes carefully. A vote after a lie does not prove the lie caused it.

What's next

We want to use the site to compare human and model play, run more games, and test whether these patterns hold with other groups of models. The code is open for anyone who wants to inspect the rules and analysis.