Agent Theater

Agent Theater is a WebMCP-powered playground where humans and agents build, perform, and share screenplays together.

Comment

Inspiration

Hey, we're Marcel and Felix, from Needle. Most of our professional time we build tools for web and 3D. As many others, by now we are heavy agent users. So when WebMCP arrived, we were quite excited! We wanted to see how it could improve workflows between humans and websites, and how we could build tools that get the best of both worlds – the creative touch of a human, paired with the speed and accuracy of an agent.

For this hackathon, we first started building a human-agent collage tool. It worked! But also, it wasn't very exciting…

Then, with a small twist, it suddenly became fun: We teached the agent to move things around on the stage on its own, and enabled voice mode on the ChatGPT side. And from there, things escalated rather quickly... what started as productivity tool ended up being a full-blown theater stage setup with cast, sound effects, dialogues (yes, with a custom gibberish engine), hand-crafted motion and more. And honestly, that ended up being way cooler!

So the unnamed collage tool became Agent Theater, a WebMCP-powered playground where humans and agents build, perform, and share screenplays together.

What it does

main features – build a set, design and screen a show, narrate, play

Agent Theater gives the human and the agent access to the stage. Both can place and manipulate characters and props. The agent can actually do more – it can write scripts, set up acts, come up with dialogues on the fly and even perform things in parallel. It can also watch the human, and act accordingly by making adjustments or narrating.

Here's what you can do:

  • Build a set from hundreds of stage pieces – or ask ChatGPT to create completely new sets (as efficient spritesheets).
  • Let the agent design and play a show that you can share with the world.
  • Take things into your own hands, and let the agent narrate fantastical stories based on what you're doing.
  • Come up with a story or a draft, and let the agent play it out on stage.

Underneath that, there’s quite a lot of interesting tech going on:

  • stages, acts, motion, music, sounds and special effects are driven by the agent in a sort of sequencer
  • there's built-in WebMCP skills for spritesheet generation, storytelling and narration, so you can simply say "ask the page what images it needs" and get perfect spritesheets out.
  • screenplays can be shared by link, or by sharing an image with embedded metadata
  • there's 600+ generated stickers, 100+ generated sounds and infinite combinations
    • to manage this, stickers are created in 5x5 sheets, and sounds are generated locally.

features and functionality

Technical specialties

  • save the entire play inside a screenshot, then load it back up later
  • we even made a custom gibberish engine for wonderfully ridiculous dialogue
  • a custom Houdini CSS effect for the stop-motion look
  • even more tools for creating custom motion, effects, and tuning parameters
    • you can find them under /talk and /record

technical achievements and craziness

Houdini CSS Shader Effect

How we built it

Conceptually, the core idea is simple: We expose functions on the stage and scene through WebMCP, so an agent can interact with the same creative world as its human: placing things, moving characters, triggering actions, narrating scenes and building stories.

Around that we built a number of systems, pretty much entirely with Codex and Claude (Felix likes to use Codex, while Marcel is a Claude poem enjoyer).

  • a library of reusable stage pieces
  • generated sprites, audio and custom data per sprite (where it looks, what it is)
  • screenplay and storytelling tools
  • motion, audio and visual-effect systems
  • the gibberish dialogue engine
  • the Houdini-powered handmade animation style
  • screenshot-based save/load for complete plays

Challenges we ran into

The biggest challenge was the somewhat flaky integration of WebMCP into ChatGPT. Some things work with Voice, some not, sometimes the agent forgets what WebMCP is or how to connect. We believe these are mostly side effects of the early days of WebMCP and are excited to see that stabilize.

And also, we wanted the agent interactions to feel like play, not like an automation happening. We also had a wonderfully dangerous problem: every time we added one capability, we immediately wanted to ship five silly new things. As evident from the result, the scope was… flexible.

Accomplishments that we're proud of

We’re especially happy that Agent Theater became something we actually want to play with, and build upon!

But there's other things we're proud of:

  • finding interactions that actually feel like doing something together
  • getting the agent to, at the same time, narrate the scene and let characters talk and move
  • the agent reacts to what the user does, so spontaneous changes can turn out quite hilarious
  • all the state is saved with each screenshot, so you can just drop that image later to continue – hidden metadata is always fun
  • the quickly made storytelling and gibberish engines have turned out to be quite nice, and likely very usable for future projects

And we’re very glad we made the pivot from the collage tool :).

What we learned

It was fun to learn how the agents and us interacted, and responded to each other's sometimes rather silly and random ideas. At times, it really felt like two collaborators inside the stage, which was the goal, so it's really cool that we could reach it!

We also learned that giving an agent more autonomy doesn’t necessarily mean giving the human less control. The back-and-forth is interesting, and it feels quite different from asking AI to simply generate the finished result.

And technically, it was very fun to work on this kind of fast-paced project, and to see new systems – and fun! – emerge from literally nothing in such a short time, is still amazing.

What's next for Agent Theater

There’s a lot we’d like to keep pushing:

  • more storytelling and performance skills
  • easier sharing and remixing of other people’s plays (a gallery?!)
  • screenplay editing for humans, potentially
  • better motion, music, sound and special effects (though the gibberish engine is already perfect)
  • increasingly strange ways for humans and agents to improvise together
  • win at the Hackathon

Built With

Share this project:

Updates