Plinky : Doodle Symphony for Kids

Plinky turns children’s drawings into playable music, removing the barriers that make kids quit music before they begin.

Comment

Inspiration

In many families, learning music still requires expensive instruments, private lessons, and structured training. Studies(WVS 2025, 51 countries and c.74k participants) consistently show that participation in arts education drops significantly among lower-income households, limiting access to creative development.

For countless children, creative expression doesn’t fail because of lack of imagination; it stops before it begins.

Plinky removes that barrier.

Instead of requiring premium hardware or motion sensors, Plinky turns any surface into a generative musical interface. A child’s drawing, a spontaneous gesture, even abstract marks can become structured sound in real time.

By shifting environmental interpretation and musical reasoning to Gemini 3’s cloud intelligence, Plinky delivers a high-fidelity, adaptive music experience on any standard mobile device. No specialized equipment. No prior training.

Plinky is not only about access.

It reimagines how creativity begins, transforming imagination into music before cost or privilege determine who gets to participate.

What it does

Plinky turns children’s drawings and physical marks into playable musical instruments. Not only ordinary paper, but any surface where contrast exists can become music: sand, flour, spaghetti, or even a clenched fist. It enables children to make music instantly through play and movement, without instruments, lessons, or setup.

Core Experience: Play → Sound

Scan & Play

By tracking hand movement in real time, Plinky maps shapes and gestures to sound, allowing children to explore music through play rather than instruction.
Drawings and physical marks become interactive instruments that respond immediately to movement and touch.

Creative Expansion: Imagine → Redesign

Explore: Magic Instrument Creator

Natural language input is transformed into reimagined instrument shapes using the Imogen API, enabling children to visually redesign instruments through text-driven imagination.

Instant Magic

Children can skip scanning and jump straight into ready-made instruments with preset hit zones, allowing immediate play and exploration.

Reflection & Continuity: Play → Meaning

AI Studio Recap

A kid-friendly performance recap featuring playful critic quotes, style labels, and recommended tracks that reflect each child’s unique play style.

AI Music & Album Cover Generation

Plinky generates responsive musical layers in real time, transforming spontaneous gestures into coherent rhythms and melodies that evolve with each interaction, along with a customized album jacket.

Your Jam & Gallery

Every session can be saved as a unique musical piece, preserving both the sound and visual identity of the child’s creation.
A shared gallery allows children and families to revisit, explore, and celebrate past creations, fostering continuity and creative ownership.

Control & Care

Settings

Provides simple controls for parents and educators, including default instrument selection, hearing guard, privacy options, content management, and expert analytics tools.

How we built it

Plinky was built around a bold but simple question:

What if making music started with imagination and movement, not instruments or instruction?

From the beginning, we treated Plinky not as a music app, but as a real-time translation layer - one that continuously converts visual marks and physical gestures into sound, with no setup, no latency, and no prior knowledge required.

Designing for Immediacy

The core design principle was immediacy.
If sound does not respond instantly, play breaks.
If feedback lags, curiosity fades.

To preserve a sense of flow, the system was designed so that every interaction produces audible feedback within a single perceptual moment. This requirement shaped all technical decisions that followed.

From Any Surface to Sound

Visual Understanding & Reasoning

Gemini 3 Reasoning

Plinky uses image understanding and contrast-based interpretation to transform drawings and physical marks into interactive zones, regardless of medium. Paper, sand, flour, spaghetti, or even a clenched fist can become a playable instrument, as long as contrast exists.

By strategically utilizing Gemini 3’s free-tier quota for the initial environmental scan, we’ve eliminated both hardware and operational costs, ensuring the service remains accessible to everyone.

Turning Movement into Music

Gesture Tracking & Playability

MediaPipe

Low-latency finger and hand tracking powers real-time gesture recognition, enabling precise mapping between movement and sound.
This ensures that musical output feels directly connected to the child’s body, reinforcing a sense of agency and control.

Generating Music That Listens Back

Real-Time Music & Visual Generation

Google Lyria
Plinky generates adaptive musical layers in real time, transforming spontaneous gestures into evolving rhythms and melodies that respond to how children play, not just what they play.

Gemini 3 Image Generation
Each session also produces a dynamic album cover, visually capturing the character and energy of the performance as it unfolds.

Music and visuals evolve together, forming a coherent artifact rather than a random output.

Expanding Imagination Through Language

Creative Exploration & Personalization

Google Imogen

In the Explore – Magic Instrument Creator, natural language input is used to regenerate and adapt instrument shapes.
Children can describe imaginary instruments in words, then see those ideas transformed into playable visual forms, bridging language, imagination, and sound.

Inspiring Discovery Beyond the Session

Discovery & Inspiration

YouTube Music API

Based on each child’s interaction patterns and play style, Plinky recommends related music to encourage listening, discovery, and creative curiosity beyond the immediate experience.

Character, Story, and Emotional Tone

Design & Storytelling

Figma
Plinky’s red monster mascot was hand-designed in Figma to give the system a friendly, memorable personality that resonates with children.

Google Flow
Illustration sequences in the demo video were AI-generated using Google Flow, then hand-edited to refine pacing, clarity, and emotional tone balancing technical explanation with narrative warmth.

The Core System Loop

At its heart, Plinky runs a continuous loop:

See → Move → Hear → Respond

Every interaction feeds back into sound with minimal delay, reinforcing play, curiosity, and creative confidence.
This loop is what allows Plinky to lower barriers not by simplifying music, but by changing how it begins.

Challenges we ran into

The most difficult challenge was not only recognition accuracy, but also responsiveness. Even small delays break immersion, especially for children. Designing reliable fallbacks across diverse materials and lighting conditions was equally critical.

Accomplishments that we're proud of

We built a fully playable system that works across paper and unconventional materials, requires no setup or instruction, and feels intuitive within seconds. The experience is fully responsive, working across devices without breaking layout or interaction, while supporting diverse accessibility needs. Integrating multiple real-time AI systems, custom design assets, and hand-edited media while maintaining stability and low latency was a major achievement.

What we learned

Working on Plinky allowed us to step away from day-to-day production tasks and experiment with new ideas more freely. We gained hands-on experience with Gemini vibe coding, learning how to prototype and iterate with AI-assisted development rapidly. The project also taught us how to collaborate more effectively, particularly around repository structure, version control, and coordination in a shared codebase.

What's next for Plinky : Doodle Symphony for Kids

Next, we plan to expand instrument diversity, enable collaborative multi-child play, and introduce adaptive soundscapes that evolve with each child’s interaction style. We also aim to add notation support, allowing children to transition from free play into performing complete musical pieces.

We plan to further enhance the Magic Instrument Creator by leveraging the Imogen API, a paid service during development, enabling instruments to take on diverse, imaginative forms in real time based on typed prompts.

Beyond Plinky, we plan to extend the ecosystem by connecting families to physical instrument vendors and online music teachers, supporting a seamless path from playful exploration to real-world musical learning.

Lastly, Plinky isn't just an app; it's a holistic educational ecosystem. We designed physical toolkits to ensure the best AI reasoning environment and to bridge the gap between digital music and tactile play, transforming every physical interaction into a symphonic discovery.

Built With

Share this project:

Updates