Inspiration

What if a child could build the beginning of a story with their hands—and hear their imagination take shape around them?

LEGO Braille bricks inspired us to connect tactile learning with collaborative storytelling. Physical letters become words, spoken ideas give those words meaning, and images and sounds bring them into a shared world.

We designed Story Bricks for blind and low-vision children, with the goal of helping blind and sighted friends create together. Touch, conversation, and spatial sound form the core experience, while visuals offer another way to enjoy the same story.

What it does

Children arrange LEGO Braille bricks into words. A camera recognizes each captured word, and a conversational voice asks questions that encourage the child to develop its meaning.

Each accepted word triggers the generation of a corresponding illustration and sound effect. The image appears at the word’s captured location, and the sound plays from that same point in space. These elements arrive as they become ready, allowing the conversation to continue while the child’s world takes shape.

For example, “RAIN” can become an illustration accompanied by rainfall sounds, while a question such as “What happens when the rain begins?” invites the child to shape the adventure.

The experience includes:

  • New Story — Create Your World: Children build words, answer spoken questions, and develop a personalized story that is narrated back to them.
  • Saved Story — Judy’s Story: Children can experience and replay a prepared story with illustrations, animation, narration, and synchronized sound.
  • Tutorial and guided introduction: Spoken onboarding introduces the starting gesture, followed by guidance for building and capturing the first word.

Story Bricks centers on camera input, touch, and audio. This interaction model offers a path toward camera-equipped AR glasses, including devices without displays. Visual content can enrich the shared experience wherever a display is available.

How we built it

We built a Unity application connected to Python services, using Meta Quest as our prototyping platform.

For recognition, OpenCV, OCR, and six-dot Braille analysis process the captured word region. Printed-letter and Braille recognition run in parallel, allowing the first complete, accepted reading to advance the interaction. Uncertain captures can prompt another attempt.

OpenAI APIs support question generation, speech transcription, story generation, and narration. Guided questions connect the child’s physical words with their spoken ideas, keeping their contributions central to the story.

For visual generation, we connected Comfy Cloud workflows that create an illustration for each word and remove its background to produce a transparent image. Unity places the illustration at the word’s captured position.

ElevenLabs generates corresponding sound effects, which Unity plays through spatial audio from the same location. Image generation, sound generation, and conversation run independently, so children can keep speaking while new elements appear.

We also integrated an optional story-video workflow that uses the generated word illustrations as references for an animated retelling.

Challenges we ran into

Reading physical words reliably. Small printed letters and raised Braille studs become difficult to distinguish under shadows, reflections, motion blur, and unusual camera angles. We developed complementary recognition paths and checks for uncertain results.

Keeping media connected to physical locations. The camera may move before an image or sound finishes generating. We needed to preserve each word’s captured location and attach its media to the correct place.

Coordinating asynchronous systems. Recognition, conversation, illustrations, and sound effects finish at different times. We worked on independent processing and clear feedback so the experience could continue while results arrived.

Making interaction understandable through sound. Spoken guidance, listening cues, and conversational pacing needed to communicate what was happening without requiring constant attention to a screen.

Accomplishments that we're proud of

We connected tactile word-building with generated images, spatial sounds, and personalized storytelling.

We developed parallel recognition methods and a conversational flow that gives children an active creative role. Their physical words introduce possibilities, and their spoken answers shape the adventure.

We also brought together live story creation, a replayable example story, and voice-guided onboarding—giving participants a way to begin, create, and experience the result.

What we learned

We learned that accessibility influences the entire interaction: how someone starts, knows a word was recognized, understands when to speak, and experiences the finished story.

We also learned that responsiveness depends on the whole system. Allowing conversation to continue while images and sounds are generated can matter as much as making an individual processing step faster.

Recognition uncertainty is another part of the experience. Requesting a new capture can protect the child’s intended word and prevent an incorrect reading from changing the story.

Most importantly, touch, sound, and imagery can provide different ways to participate in the same imaginative activity.

What's next for Story Bricks

Animate individual words in place. Beyond the completed story-video workflow, we want each word’s illustration to become an animated character or object at its physical location, making the world more expressive during creation.

Make capture more natural. The current interaction uses a hand gesture to initiate capture. We want to explore automatic capture when children move their hands away from a completed word, with checks that avoid capturing an unfinished arrangement.

Improve recognition accuracy and latency. We plan to investigate more reliable Braille and printed-letter recognition under poor lighting, blur, occlusion, and challenging viewing angles, while reducing processing delays.

Extend the experience to lighter devices. We want to adapt the camera-and-audio interaction to AR glasses, including displayless devices, and evaluate the experience with blind and low-vision children and educators.

Built With

Share this project:

Updates

Submission history