Curtain Call AI

Gemini 1.5 Pro directs miniature magicians in a unique real-world stage show! It writes, designs, and animates, proving the power of generative AI to create magic! 🤖🧠✨

Comment

Potential Value to Businesses

Curtain Call AI shows how Gemini can be used to orchestrate a unique customer experience. In this example, Gemini creates a mini-animatronic show that changes every time you watch it. That by itself has value, but imagine the possibilities when you can orchestrate a customized journey both online and in-store.

Potential Community Impact

Curtain Call AI could have a huge impact on STEM education. It can bring together young artists, engineers, and AI enthusiasts. It can provide a physical element and performance to the classroom to inspire students. I envision a mini-theater on wheels (like a puppet theater) but with a WiFi connection and drawers full of LEDs, servos, and various other widgets that could be easily swapped out to modify the show.

Inspiration

I was inspired by a few things:

  • I saw a Blue Man Group performance in Vegas and started thinking about all the work that goes into creating a production like that. I also saw a few snippets of the Taylor Swift Eras tour on Disney and thought about all the visual effects programming that must go into creating such an extravagant show. This includes the writing, directing, lighting set design, etc. I wondered if generative AI is advanced to choreograph all the different elements of a simple stage show.
  • I saw a mini autonomous car exhibit at the Google Next conference that leveraged Gemini and onboard computing to navigate to their destination.
  • There was also a Train to Cloud City exhibit at Google Next that used a Raspberry PI and RFID to teach folks how to architect GCP solutions. Both of these exhibits got me excited about doing some hardware hacking.
  • I have always admired the work of Disney Imagineers, who create animatronic shows and rides that implement creative elements, such as the Haunted Mansion. I wondered if we could use generative AI to make every show or ride unique.

What it does

Curtain Call AI is a set of NodeJS scripts that leverage the Gemini 1.5 Pro API to create a unique comedy stage production involving two miniature magicians.

How I built it

I used Visual Studio Code to build the NodeJS scripts and heavily leaned on Google AI Studio for the initial feasibility testing and code generation.

Architecture and Software

Gemini handles many aspects of the show creation process, including:

  • Writing the character scripts
  • Designing the lighting given a DMX lighting configuration in JSON
  • Writing Imagen 2 prompts to create the set background images
  • Picking which sound effects from the JSON library document to play
  • Choreographing character movements via RC servos

The output from Gemini is a JSON script document with various cues, including lights, delays, script lines, sound effects, background changes, and animations.

After Gemini produces the show and outputs a JSON script, the NodeJS generation app uses the Google Text-to-Speech API to generate speech audio and Imagen 2 to generate backgrounds.

Once the show is generated, the run-show.js script is used to run the show. It parses through each cue in the JSON script and runs it using a few different outputs:

  • Script lines and sound effects - The cmdmp3.exe command line audio player by Jim Lawless plays each line and sound effect.
  • Lighting - The NodeJS enttec-open-dmx-usb library by Moritz Ruth is used to interface with an Enttec Open DMX USB interface to control the DMX lighting.
  • Scene Background - A computer monitor serves as the stage background, and IrfanView by Irfan Skiljan is used to display the Imagen 2 generated images (16:9) in full-screen mode.
  • Character Animations - The NodeJS Johnny-Five library, by Rick Waldron and other contributors, is used to control servos that rotate each character via an Arduino running Firmata.

Physical Components

Stage - The stage is built from scrap wood and held together with hot melt glue and finishing nails. I spray-painted it matte black to keep focus on the characters.

Servo Animation - The servos that turn the toy dolls are standard hobby servos recycled from a previous project and mounted to the stage with screws.

Toy Dolls - The toy dolls came from Walmart and are secured to the servo horns using screws and hot melt glue.

Lights - The DMX lighting fixtures are recycled from a couple of previous projects and connected to the Enttec Open DMX USB interface via the same DMX cables you might find in a full-size stage production.

Arduino - Arduino is recycled from previous projects and runs Firmata. It is powered by an external battery to ensure the connected servos don’t draw too much current from the USB bus.

Challenges I ran into

I hit a few snags:

Wrong model, silly!

The output was terrible when I started generating the show script via API calls instead of Google AI Studio. The JSON was poorly structured, and it would often output my example JSON instead of generating a new show. It took me a while to realize that the model string in my getGenerativeModel() call was “gemini-pro” and not “gemini-1.5-pro-latest.” Needless to say, the output from Gemini 1.5 Pro is way better! Once I fixed the model string, the API output quality matched what I saw in Google AI Studio.

How do I call the Imagen API from my JavaScript code?

I could find an example showing me how to call the Imagen/Image Generation API from JavaScript. Fortunately, there was an example using Curl. I asked Google AI Studio to generate a JavaScript version based on the Curl request it did a pretty job. I only had to correct 2-3 lines to get it working.

Johnny-Five library issue

I struggled to get Johnny-Five to talk to my Arduino on one of my test machines. It turned out that two of the libraries Johnny-Five had marked as optional, "serialport" and "firmata," were not actually so optional and silently failed to build during the install. Once I figured out the problem, I just had to update my Python version, run "pip install setuptools," and then reinstall Johnny-Five so binaries would properly build. It worked fine after that.

Accomplishments that I'm proud of

Demonstrating how Gemini is now sophisticated enough to direct a relatively complex creative process.

What I learned

I learned how to leverage Gemini for a creative process by providing it with assets and configurations via JSON documents. This allows me to generate consistent output that can be consumed by another process without human intervention.

What's next for Curtain Call AI

  • Improve the text-to-speech output somehow so it sounds more natural and expressive.
  • Animatronic mouth movements so you can easily tell which character is speaking each line.
  • Add other visual and mechanical elements to the production.
  • What other creative processes can Gemini control? Here are some ideas I have:
    • Holiday Lighting
    • Designing and 3D printing sculptures
    • Craft stories based on a scan of my kids’ artwork.

Built With

Share this project:

Updates