Embodied Blocks

Inspiration

Tangible play is how children learn about the world. Memory-building is how they make meaning. And storytelling is how humans connect. Our toy orchestrates all three—without inserting screens into the sacred space of childhood.

Embodied Blocks Website

What it does

Embodied Blocks is a modular toy that deepens a child's engagement with the physical world through sense-based reflection and storytelling. Children have the agency to manipulate building blocks, reconfiguring not just the toy’s form, but its physical intelligence as well. Through the addition of sensor blocks, the child gives the toy eyes, ears, a body that provides knowledge about the scenario being acted out as they interact with it in the real world to create a story.

Our approach separates play from processing. After the toy is put down, it generates a story narration according to its inferred perception of it depending on its sensor capabilities. This is intentional to encourage reflection instead of interfering with active play: children gain the ability to learn from and share records about their creative, sensory exploration, and parents gain a window into their child’s world through their play information.

How we built it

Hardware

Raspberry Pi 5 (or Zero 2W)
Raspberry Pi Camera Module 2
Mic (INMP441)
IMU
Bluetooth Speaker

Setup

Python 3.10.x
Flask server on local machine
Claude API Key
OpenAI API Key
Hume API Key

The Raspberry Pi 5 uses IMU movement as the trigger to record sensor data. Raw data collected from any of its sensors (camera, mic, etc) is converted to a usable filetype (eg. MP4) then streamed to a Flask server running on a separate local machine. The audio/video data is proccessed with the following APIs:

Audio Sentiment -> Hume
Audio Transcription -> Whisper
Video Transcription -> Claude Vision

Next, the sensor data is sent as a string to Claude (Haiku 4.5) along with any previous story data and a custom system prompt that generates a short narrative. The returned narrative is transcribed into speech using OpenAI and played through a bluetooth speaker in the toy.

Usage

Upload fullTest.py to Raspberry Pi and run it. python fullTest.py`
Run server.py on a local machine. python server.py

Challenges

One challenge was dealing with missing sensor data, in the case where a child chooses not to build the toy with certain sensor blocks. We edited the system prompt to treat this as a feature, not a bug:

Claude's narrative talks about how quiet the world is and focuses on more visual descriptions when there's no mic.
Claude's narrative describes the ambience of the scene in a dark, muted environment when there's no camera.
Overall, Claude does take the audio transcription as the biggest factor in story generation, as it is likely the child is also narrating what they intend to be happening in their scene. To ensure the child's creativity is the driving force for the narrative, Clause does not make up anything it can't physically sense or isn't told about.

Another challenge was deciding when to start and stop a play session. We didn't want the toy to react live to the scene, because our goal was not for children to interface directly or conversationally with an AI. We wanted the AI to act more as a passive reflection tool, so that children could still engage in active play where they are interfacing with the tangible world using their senses as a means of creating the narrative scenario. We decided to use the IMU as the core trigger: the toy starts a play session when it detects movement past a certain threshold, and ends play and recording once it's put down for a set time.