Inspiration
There's a moment in Jumanji: Welcome to the Jungle where Dwayne Johnson's character punches a random background NPC, testing a theory that if a character isn't important to the story, they're not really "there." It's a throwaway joke, but it made us laugh way harder than it should have, and it stuck with us. It's also built on something true: every movie, every game, every crowd scene is populated by extras running on fixed, duplicated scripts, waiting to say their one line or walk their one loop.
That's not a joke to the people paying for it, though. Believable crowds are either expensive (real extras, mocap, hand animation) or obviously fake: the "clonethroat" effect of the same faces looping through a scene, background characters with no micro-behavior standing around like mannequins, NPCs strolling past an explosion because nothing told them to react. So we started digging into the actual question underneath the joke: what if the fix wasn't better art, but better cognition? What if every background character had an actual mind, cheap enough to run by the hundreds?
What it does
NPC-Neural runs a live 14-character noir scene, a confrontation in a manor bar. Two principal leads (a detective and the suspect he has just accused) get rich, frequent reasoning ticks. Twelve background extras (bartender, waiters, party guests, a pianist, security, a photographer, a valet, a coat-check attendant) get cheap, infrequent ticks that escalate to a fuller reasoning pass when something notable happens near them.
Every tick writes the character's context, reasoning, decision, action and mood to ClickHouse as it happens. The web frontend shows the whole cast live. Click any character to see the goal it set for itself, its mood, the action it took and the reasoning behind it, all read back out of the database. Or ask the Director's Assistant a question in plain language and get an answer grounded in a live query rather than a cached summary.
Three things make it more than a visualisation:
Emergent chain reactions. Characters react to each other's real generated actions, not to scripted triggers. In one recorded run, a guest ducking behind a decorative urn rippled through five characters in ninety seconds. A waiter rewrote his own goal to "keep an eye on the urn while waiting for the path to clear". Security shifted to "look past the coat-check counter to see what is happening by the exit doors". A second guest copied the ducking. A fourth set out to "figure out why that person is sneaking away". Nothing in the codebase wrote that chain, or that phrase.
Crowd scale on demand. A menu scales the cast from 10 to 500 synthetic NPCs writing decision rows continuously, so you can watch the write volume climb while the crowd-wide mood aggregate still answers in well under a second.
A console page exposing the live database connection, the MCP tools every query is routed through, the deployed schema with real row counts, and the rows landing right now.
How we built it
Each NPC is an independent agent built with Google's Agent Development Kit
(ADK), the open-source framework behind Agent Builder, running on Gemini
(gemini-3.5-flash for principals, gemini-flash-lite-latest for the
background cast). Every reasoning tick feeds the agent its current context
and goal and returns a structured decision: an action, a mood, its reasoning,
and optionally a rewritten goal. A second ADK agent answers freeform
questions using run_query and list_tables as tools.
ClickHouse Cloud holds the simulation's memory. The access pattern is simple
and steady: the agents produce an append-only stream of context ticks,
decisions and events, and almost every read is either "what is the latest
state of each character" or "what is true across the whole crowd right now".
Four MergeTree tables ordered by (npc_id, ts) cover both. Latest-state
lookups use argMax(...) grouped by character, which is what fills the
inspector panel, and the crowd view is one aggregate query over the whole set
rather than sending per-character rows to the browser. During a 500-NPC scale
run, around 190,000 rows in, that aggregate was still returning in roughly
half a second while writes continued. That is a small workload by ClickHouse
standards, but it is the behaviour the scale feature depends on.
Every read and write goes through the official mcp-clickhouse MCP server
over stdio. There is no direct ClickHouse driver anywhere in the codebase.
Two MCP sessions are held open for the life of the process with separate
privileges: read-only for the scene, write-enabled only for the scale test.
A tiering layer sits between the agents and Gemini, scheduling frequent full-context reasoning for principals and infrequent ticks for background characters, and escalating a background character to a richer pass when something notable happens to it. The scheduler is deterministic interval math and never makes a model call of its own.
A proximity graph determines which characters can perceive which others, so reactions propagate along real spatial lines instead of globally. It decides only who could plausibly notice, never what they do about it. The reaction is entirely the receiving character's own next Gemini call reading the source character's real generated action text, which is how multi-hop chains emerge without any script writing them.
The frontend is FastAPI serving a vanilla JS canvas. Idle characters are ambient points of light connected by the real proximity edges, and when an event ripples through the crowd, light travels node to node along those edges at a visible speed instead of flashing everywhere at once.
Challenges we ran into
ADK's McpToolset versus mcp-clickhouse. The installed mcp-clickhouse
release requires mcp>=2.0, while ADK's built-in McpToolset requires
mcp<2. This is a genuine, unresolvable version conflict. We worked around it
by wrapping the same MCP ClientSession as plain ADK function tools instead
of using McpToolset. It is still the real server over the real protocol,
just not that one wrapper class.
Free-tier Gemini quotas are much tighter than the published numbers.
Confirmed by live 429 responses: gemini-3.5-flash capped at 20 requests per
day, and the lite model at 15 per minute and 500 per day, per project. With
twelve background characters, a naive scheduler produced bursts that blew the
per-minute cap immediately. We fixed it with staggered tick starts, spreading
each tier's first tick across its interval instead of firing everyone at
once, plus a shared token-bucket limiter used by every lite-model call, so
the app throttles itself proactively rather than just reacting to failures.
Making the crowd view mean something. An early version of the scale test laid the synthetic cast out on a grid and coloured it straight from the aggregate mood counts, which meant each mood painted a solid horizontal band. It looked like a spreadsheet, and worse, it implied an ordering that did not exist. We reworked it to scatter positions deterministically and to read each dot's name and mood back per character, so what you see corresponds to actual stored rows.
Graceful degradation as a design goal. Once the daily principal quota is exhausted, principal characters transparently drop to the background model instead of failing outright, so the simulation and the Director's Assistant keep working for the rest of the process.
Accomplishments that we're proud of
The chain reactions are the thing we keep coming back to. We did not write a ripple system. We gave each character a way to perceive a neighbour's action and left the response entirely to that character's own reasoning. Watching a five-character cascade come out of one guest ducking behind an urn, including a goal none of us would have thought to write, was the moment the project stopped feeling like a demo.
We are also proud of the discipline around data access. It would have been faster to reach for a ClickHouse driver directly and mention MCP in the README. Instead every single read and write in the project goes through the MCP server, including schema creation, which means the integration is demonstrable at runtime rather than merely claimed. The console page exists so anyone can verify that in about ten seconds.
And the quota engineering, unglamorous as it is, is what makes the thing run at all on a free tier.
What we learned
Free-tier LLM quotas are a first-class design constraint for multi-agent work, not an edge case to patch over later. Treating exhaustion as an expected runtime condition, with staggering, rate limiting and graceful model fallback, mattered as much to whether the system felt finished as the reasoning quality did. The same architecture would simply run with a much higher ceiling on paid quota.
The other lesson was about where to draw the line between authored structure and generated behaviour. The proximity graph and the notability check are authored. They decide whether an action is noticeable and who is near enough to notice it. They never decide what happens next. Keeping that boundary strict is the entire reason the chain reactions are believable rather than a puppet show, and it is a useful principle well beyond this project.
Finally, the storage layer shaped what we could put in front of a judge. Because the crowd-wide question stayed cheap to ask while writes kept landing, the scale view could show a live aggregate rather than a precomputed chart.
What's next for NPC-Neural
- Swap the fixed demo cast for a scene-description-driven cast generator, so a director can type a setting and get a populated, reasoning crowd.
- Persist and replay full scenes for previz review, including scrubbing back to any moment and asking why a given character did what it did.
- A richer scene view with movement and animation rather than fixed positions.
- Longer-horizon memory per character, so a background extra can carry a grudge across a whole scene instead of reasoning from a rolling window.
Log in or sign up for Devpost to join the conversation.