You see the bomb. Your AI agent holds the manual—as WebMCP tools. Neither of you can defuse it alone.
Inspiration
Most AI experiences revolve around a chat box: the human asks, the agent answers, and one side usually has all the important context. We wanted to explore something more physical, cooperative, and playful.
CROSSTALK began with a simple question: What if WebMCP was not merely an integration layer, but a core game mechanic?
We drew inspiration from cooperative bomb-defusal games, escape rooms, radio communication, and the tension of working with someone who knows things you cannot see. In CROSSTALK, the human and AI agent are deliberately given different abilities. The human can see colors, symbols, gauges, displays, and flashing signals. The agent has the operating manual, machine-readable data, and control tools—but cannot perceive the device itself.
Success comes from communication, not automation.
What it does
CROSSTALK is a cooperative browser game in which the second player is an AI agent.
The human player is presented with a physical-looking device containing modules such as:
- A wire bay with colored wires
- A keypad covered in unfamiliar glyphs
- A regulator with an analog gauge
- An echo module with changing labels and displays
- A signal transmitter producing short and long beeps
The AI agent receives a changing set of WebMCP tools. These let it consult the manual, scan machine-readable identifiers, inspect recorded history, adjust controls, and review completed sessions. However, the tools intentionally do not reveal information that only the human should perceive.
The resulting loop is:
Describe → Look up → Decide → Confirm → Act
For example, the agent may know exactly how to decode a signal frequency but must ask the human what rhythm they hear or see. The human may know where the gauge needle is, while only the agent can operate the regulator. Neither participant has enough information or control to finish alone.
CROSSTALK contains three escalating missions:
- HANDSHAKE introduces communication through the wire bay.
- CROSSED WIRES combines wires, glyphs, and regulator controls.
- SILENT FREQUENCY tests the full partnership with keypad, echo, signal, and wire modules.
Every agent action is shown in the in-game TEAM RADIO, so the human can see what the agent inspected or changed. Mistakes cause strikes, while successful coordination produces a detailed debrief and training record.
How we built it
CROSSTALK is a client-side TypeScript application built with Vite and the WebMCP API.
We treated the WebMCP tool schema as a sensory boundary. Each tool defines not only what the agent can do, but also what it is allowed to know. Tool descriptions explicitly explain missing sensory information and direct the agent to ask the human instead of guessing.
The game currently exposes 11 imperative tools and one declarative form tool. These include manual lookups, device inspection, RFID scanning, regulator controls, echo-history retrieval, transmitter tuning, and post-mission review.
The toolset changes dynamically throughout a mission. When a module becomes active, its relevant tools appear. When that module is solved, those tools are removed using scoped AbortController lifecycles and a toolchange event is emitted. Read-only sensors and irreversible actuators are also distinguished through tool annotations.
To prevent puzzle logic and documentation from contradicting each other, the operating manual and the actual module behavior share the same underlying rule data.
We built the interface as a physical workbench rather than a conventional web dashboard. Mission case files, taped instructions, a clipped briefing docket, radio receipts, paper textures, switches, lamps, and a ring-binder manual make the experience feel like handcrafted field equipment. Audio is synthesized with WebAudio, while the visual assets and textures are original, code-native work.
Everything runs without accounts, analytics, or a game server. The local operator dossier records only relevant game events—not the player’s conversation—and remains on the device.
Reliability is covered by Vitest and Playwright, including:
- 27 automated tests
- Deterministic zero-strike runs through all three missions
- Native WebMCP testing in Chromium
- Firefox and WebKit compatibility checks
- Accessibility and interaction checks
- Responsive layouts at desktop, mobile, and 320-pixel widths
- Reduced-motion, keyboard, and forced-color support
- Production deployment verification
Challenges we ran into
The hardest challenge was creating information asymmetry that felt fair.
If the agent received too much state, the human became unnecessary. If the agent received too little, the interaction became frustrating. We had to decide carefully which facts belonged to human perception, which belonged to machine-readable sensors, and which actions should require explicit confirmation.
Dynamic tool registration created another challenge. Mission resets, completed modules, and stale browser sessions could otherwise leave the agent with tools that should no longer exist. We built lifecycle management around each tool and added tests for repeated missions and state transitions.
We also spent significant time making the interface feel authored rather than generated. Early versions resembled a collection of polished panels, but not a believable game. Moving toward a cohesive physical workbench—with imperfections, paper layers, hardware controls, and mission-specific case files—gave CROSSTALK a much stronger identity.
Human playtesting caught an especially useful issue in the final mission: the signal module’s beeps could be heard, but its visual feedback was too subtle to notice. Automated logic tests passed because the signal state was technically correct, yet the experience was still unclear. We redesigned it with a large synchronized VISUAL BEEP lamp and a reacting speaker grille, then added rendered-state regression coverage so the problem could not silently return.
Accomplishments that we're proud of
We are most proud that the AI agent is not simply a hint system or chatbot attached to the side of the game. It is a genuine second player with unique knowledge and controls.
Other accomplishments include:
- Making WebMCP itself part of the game design
- Creating puzzles that neither participant can solve independently
- Dynamically changing the agent’s capabilities as the device changes
- Keeping every agent action visible and understandable to the human
- Building evidence-bounded coaching without recording conversations
- Creating an original, tactile visual identity without external game assets
- Supporting desktop and small mobile layouts without sacrificing clarity
- Completing deterministic, zero-strike automated runs of every mission
- Shipping the complete experience as a private, static, client-side application
Most importantly, CROSSTALK is fun to play. The tension does not come from pretending the AI is human. It comes from learning how to communicate effectively with an intelligence that perceives and controls a different part of the world.
What we learned
We learned that tool descriptions are part API documentation, part user interface, and part game design. A small ambiguity in a description can change how the agent behaves just as dramatically as changing the puzzle itself.
We also learned that clear limitations make an agent feel more trustworthy. Explicitly telling the agent what it cannot see produces better cooperation than giving it vague access to everything.
Dynamic capabilities are more understandable than one enormous permanent toolset. When tools appear only when relevant, the agent receives a clearer picture of the current situation and the player can more easily understand its role.
Finally, automated correctness is not the same as playability. State-level tests told us that the signal light worked; a person playing the game told us that it could not actually be noticed. Combining deterministic testing with real human play sessions made the experience considerably stronger.
What's next for CROSSTALK
Next, we want to expand CROSSTALK with additional missions built around new asymmetric senses and controls: spatial descriptions, timed synchronization, memory, visual pattern comparison, and situations where the human and agent must negotiate uncertainty rather than search for one fixed answer.
We would also like to add:
- More adaptive training drills based on previous mission evidence
- Optional difficulty and accessibility profiles
- Additional manual rule packs and randomized mission variants
- Richer local debriefs and replayable team-performance timelines
- New physical module designs and environmental storytelling
- Better support for comparing how different agents communicate and recover from mistakes
Longer term, we see CROSSTALK as more than a game. It can become a playful way to teach agent literacy: how to communicate missing context, verify assumptions, recognize capability boundaries, and safely coordinate consequential actions with an AI system.
The future of human-agent interaction should not be humans watching agents do everything. We believe the more interesting future is one where both sides bring something essential to the team.
Log in or sign up for Devpost to join the conversation.