Tubi the Frog

Inspiration

The inspiration for this project comes from the interactive attractions found at theme parks like Disney and Universal, where animatronic characters talk, react, and make guests feel like they’ve stepped into a story. Those moments stick with people because they’re surprising and personal, and we wanted to recreate that unique feeling.

What it does

Tubi the Frog is an animatronic wizard that holds a live and engaging conversation with the user. Through humour and imaginative hypotheticals, Tubi will determine what kind of magic user you are best alligned with.

Awaken the wizard Tap your phone on the NFC chip on the machine. It opens the Tubi website and wakes him up.
Hold the crystal ball to speak Press and hold the crystal ball on your screen and talk. It uses your phone’s native microphone, so there’s nothing to download or install.
Answer Tubi’s hypotheticals Tubi asks you a few fun, ridiculous questions. Watch him come to life as he talks, with his mouth and wand animating in real time.
Let him read your destiny After a back and forth conversation, Tubi will come up with a fortune for the user to make the interaction more memorable.

How we built it

We used 3 different mediums: MCU, micro Computer, and Laptop. The goal here was to split the workload between devices so one doesn’t take more load than the other. We used an ESP32 as our MCU, Raspberry PI 3 as our micro computer, and a website on any high level device.

Web’s job: The web would work as our way of talking to this frog, converting the speech to text using Open AI’s whisper model. The user must hold the crystal ball presented on the screen and speak to Tubi. This data that is taken from the mic audio would be packaged up and sent through websocket to the LLM on the pi. PI’s job: This was our bridge/communication. This is where we focused on hosting our website we’ve created for tubi, this would act as our remote server. Additionally here is where we linked the logic and reasoning by taking the text translated from the website and feeding it into an LLM (Open AI: gpt-5o-nano) where we then fed this response generated into a text-to-speach model (Eleven labs: eleven-turbo-v2-5). Finally the decisions made by the LLM would be packaged up with the mp3 file and sent to the ESP32 through serial connection. MCU’s Job: Here we primarily focused on using state machines and custom embedded code to handle our audio and servo movement. Additionally this is where we parsed the output of the PI’s output, handling how servo movement would work based on words provided in the package.

Challenges we ran into

This project presented many challenges along the way, forcing us to constantly rethink. On the hardware side, we had a hard time getting the audio on the speaker to work exactly how we wanted it to. In some cases it would only produce static, and others would not produce anything at all. We found that sending byte data through the same connection blocked the audio and servo would produce noise in the amplifier. Separating the UART cables gave it a chance to split the work so no data would overlap. Our biggest obstacle involved a 3d print failing, forcing us to rethink the concept in the final hours of the competition.

Accomplishments that we're proud of

We are very proud that we have a working animatronic that is very interactive. Seeing our vision come to life with some servos, cardboard, and paint made us seem like the happiest hackers out there.

What we learned

We learned that one of the most important skills when working on a project is improvisation. The first idea you have is rarely the idea that makes it to the finish line, and this project is proof. Every step of the way, our plans were challenged and we were forced to adapt. For example, a major portion of our intial idea had to be scrapped entirely within the last few hours of the hackathon. Being able to respond to such a major setback is a useful skill im glad this hackathon taught us.

What's next for Tubi the Frog

In the future we hope to give Tubi the opportunity to move into the snack industry, being able to sit on snack dispensers around the world and casting spells and choosing what snacks each person can get.