TedTime

A cozy bear that families can use to share stories with their loved ones.

Comment

Inspiration

Three out of our four group members have grandparents that live outside of the country. We wanted to create a way for families to stay connected across distance, while preserving the voices and stories of loved ones in a form that feels more personal than a phone call or simple audio recording.

What it does

Our bear serves as a way for families to share stories in an immersive way even if they may be far apart. We built a custom backend API that processes all incoming audio files to add immersive sound effects. On top of that, the bear moves its arms based on the dominant frequency of the incoming audio file, creating a fully immersive experience through sound, movement, and haptic feedback from the bear itself.

How we built it

We built a custom backend API to process all incoming audio files along with a full-stack React Native app for users to conveniently upload and share their audio files. Our backend API passes the audio file into the Deepgram API, extracting raw, unprocessed text along with timestamps for every single word said in the story. This transcript is then passed into Muse Spark along with our catalog of 100+ background sound effects, which then returns specific timestamps for when to play each of the sounds. We developed this background by using the Freesound API, extracting the most common sounds needed in kids’ stories. Our backend API offers two endpoints, one that returns a processed mp3 file with sound effects, and another that returns a json with descriptions, hashtags, as well as story images produced by Grok Imagine. Our frontend, a full-stack React Native app hosted on Expo Go, calls this API to process all our audio recordings. In addition, Grok Imagine generates image covers for different stories, making the app easier to use and more immersive for families. We used an ESP32 microcontroller coupled with a brushless motor driver to control the physical teddy bear. The audio was transferred over Bluetooth and processed locally using a Fast Fourier Transform (FFT) to extract the dominant frequencies of the speech and translate that into immersive arm movements. Using brushless motors means these movements are quiet and not overly rigid, keeping the experience friendly and fun. We used Expo for our React Native application and Supabase for authentication, data storage, and storing information about users and their stories. Recorded audio is sent through an AI backend that analyzes the story and determines where different sound effects and teddy bear interactions should occur. AI models help transform a normal voice recording into a more immersive storytelling experience while still keeping the loved one's original voice at the center of it.

Challenges we ran into

One of our biggest challenges was communication between the ESP32 and our mobile application, especially figuring out how to reliably send commands to the bear while keeping the experience simple for the user, which was why we eventually settled on Bluetooth We also ran into issues with sound effect normalization, since effects from different sources could have drastically different volumes compared with the original recording. On the hardware side, our 3D printing timeline limited how quickly we could iterate on the internal components of the bear. Our biggest software challenge was developing the system design and deciding how all our components would work together. We originally planned to create one backend repository, one frontend, and one firmware. However, our full system created a full stack app powered by Supabase along with a backend repository with its own Supabase instantiation to store our catalog of background sfx.

Accomplishments that we're proud of

We're proud that we were able to connect so many different parts of the project into one experience. TedTime combines a mobile app, AI audio processing, Bluetooth communication, embedded hardware, and a physical, animated stuffed animal rather than existing entirely on a screen. We're also proud that the technology serves a simple purpose: helping families feel a little closer together.

What we learned

We learned a lot about designing software and hardware together, especially how decisions on one side can quickly affect the other. We gained experience working with Bluetooth and the ESP32, building a React Native app with Expo, handling audio recordings and storage, and integrating AI into an actual user-facing pipeline. More importantly, we learned that adding more technology does not always make an experience better. For TedTime, the technology works best when it stays in the background and lets the storyteller's voice and the physical bear remain the focus.

What's next for TedTime

We hope to continue refining the app, gathering user opinion on how to best improve immersion and connection between families. TedTime's stuffed animal companion will also receive more expressive movements and haptic interactions that can respond directly to moments within each story. In the future, we would also like to make it easier for multiple family members to share stories with the same bear, building a growing library of voices and memories that children can return to whenever they want.

Built With

Share this project:

Updates

Submission history