The Secret of WebMCP Island

Play the classic game The Secret of Monkey Island through ChatGPT and WebMCP. Ask the agent for help, talk through the game, or just let Sol try to beat LeChuck and win the game.

Comment

Inspiration

The Secret of Monkey Island was my favourite game growing up and I've played it countless times over the years. It's a point and click adventure, with incredible humour and often tough puzzles.

I wanted to explore whether I could let an agent play a game via WebMCP and I wanted to see whether I could make that happen using Codex.

I also wanted a fun way to test WebMCP beyond a conventional website. If WebMCP can describe a web application to an agent, could it describe a 35-year-old game running inside WebAssembly?

What it does

The applications runs the original DOS VGA release of The Secret of Monkey Island in a browser using ScummVM compiled to WebAssembly. Players provide their own game files, which stay inside the browser and are never uploaded.

Once loaded, the game exposes seven WebMCP tools. An AI agent can observe the current room, see characters and objects, inspect inventory, walk to destinations, use the game’s verbs, choose dialogue responses, start a new game, wait for actions to complete, and save progress.

In my experience, the agent mostly plays through WebMCP, though I have observed it take screenshots once in a while to see where they are in the game.

How I built it

ChatGPT and I started with ScummVM and a fork that exposes semantic information from its SCUMM engine. I pinned that source revision and maintain my browser-specific changes as a series of patches.

I compiled the SCUMM engine to WebAssembly using Emscripten. A C++ bridge exposes observations and actions from inside the running engine, while a TypeScript controller manages revisions, asynchronous actions, errors, cancellation, and traces. A WebMCP adapter registers the seven tools with the browser.

I built the surrounding application with TypeScript and Vite. It validates user-selected game files before mounting them into ScummVM’s in-memory filesystem. The browser shell supports both the File System Access API and a directory-input fallback for embedded browsers.

Saved games are written through ScummVM and synchronize them to IndexedDB using Emscripten’s IDBFS. The hosted version runs as a ChatGPT Site, while the complete ScummVM and WebAssembly toolchain can also be built locally from source.

Challenges I ran into

The largest challenge was connecting a synchronous, frame-driven game engine to asynchronous browser tools. Walking, conversations, scripts, and room transitions do not finish when a command is accepted, so I needed an explicit action lifecycle and a wait_for_game tool that observes the engine rather than relying on fixed delays.

Dialogue was a pain. The original SCUMM interface reuses numbered verb slots for conversation choices, including hidden and consumed entries. I had to ensure the choices reported through WebMCP exactly matched those the engine would accept. In early testing the agent would choose an old option and exit conversations when it meant to continue learning information.

Persistence was even more fun. Calling ScummVM’s save function directly from JavaScript re-entered WebAssembly while Emscripten’s Asyncify machinery had already suspended the game loop, producing an out-of-bounds memory failure. I fixed this by scheduling saves and restores through the normal game loop, polling for native completion, and synchronizing browser storage only afterward.

Browser file permissions were another challenge. I could not bundle or redistribute the game data, ScummVM is GPL3.0, but the game remains under copyright. I wanted to use the browser File APIs to maintain consistent access to a directory with user permission, but the built-in ChatGPT browser doesn't support this API. Chrome still should, so loading the game files once should make it possible to continue when revisiting later.

Accomplishments I’m proud of

I completed a real playthrough from a new game to the SCUMM Bar, the town on Melee Island and beyond using only WebMCP tools after the initial game file selection. The agent navigated between rooms and held multi-step conversations with an ordinary pirate in the SCUMM Bar, the important-looking pirates, and even with a dog. The agent set off to master the three trials to become a pirate, selecting swordsmanship as its first challenge.

I'm genuinely proud to have worked with Codex to build an application that I wouldn't know where to start with (patching the 25 year old C++ in the ScummVM) while also learning how WebMCP works and connecting the two. All done while reviving my love for grog.

What I learned

I learned that old games contain a rich internal world. Rooms, actors, objects, verbs, inventory, dialogue, and scripts are all structured inside the engine; the difficult part is exposing that structure safely and consistently to an agent.

I also learned that observation is as important as action. Agents perform much more reliably when every action is based on a fresh revision and the environment provides an explicit signal that an asynchronous action has finished.

Most importantly, I learned that WebMCP can make an interactive experience agent-accessible without reducing it to coordinates and screenshots. The agent can reason about “talk to the pirate” or “open the door” rather than guessing where those things are on a canvas.

What’s next for SCUMMWebMCP

I would like to see whether an agent can play the entire game through on their own. Plus, I love the game, so I'd like to play it with an agent. I have experimented with voice mode in ChatGPT to direct the game, which was good, but I couldn't get ChatGPT to put on a pirate accent while playing. It did drop some extra "Arrrr" into it's vocabulary though.

One thing that didn't work initially was audio. I would like to get the audio back as the soundtrack to the game is as captivating as the pirate banter.

My next step is to broaden the experiment beyond one release of one game. I would like to support more SCUMM titles and game variants while keeping their engine-specific semantics behind a consistent WebMCP interface.

I would also like to build some explicit memory into the game for agents that are playing. Ensuring they know what they are holding and what they are currently trying to achieve would give them a better chance to win the game.

Ultimately, I would like to learn the real secret of Monkey Island.

Built With

Share this project:

Updates

Submission history