Finger Wand

• 

  Finger Wand turns a webcam and a child’s hand into an interactive learning tool.

• 

  In Melody Steps, children listen to a short melody and use hand gestures to select notes at different pitch levels.

• 

  After tracing and hearing the letter, children complete a gesture-controlled matching game.

• 

  Children pinch their thumb and index finger to trace a glowing path in the air.

• 

  The child recognises the dolphin, points toward it, and pinches to grab the card.

• 

  After selecting the dolphin, the child moves it toward the glowing Explorer Bubble and releases it to answer.

• 

  Children practise forming the letter B by moving their finger through each stroke in sequence.

• 

  The moment I saw Codex bring my hand-gesture learning idea to life for the first time. ✨

Inspiration

Hi, I’m Sarah Li, a K–12 music educator with six years of experience working with children across different age groups. I am passionate about exploring how AI can help educators create more interactive, creative, and accessible learning experiences.

For a long time, there has been a gap between the people who understand classroom needs and the people who can build educational technology. Teachers may have strong ideas grounded in daily practice, but often lack the technical resources to turn those ideas into working products. With OpenAI, Codex, and modern AI-assisted development tools, that gap is beginning to close. Educators can now translate real classroom observations, teaching strategies, and pain points into interactive software through natural-language prompting and rapid prototyping.

Finger Wand was inspired by two observations from my teaching experience.

First, young children are naturally drawn to screens and tablets, but most digital activities are designed for one child sitting alone and touching a device. There are far fewer experiences that encourage children to stand up, move, create, and participate together. I wanted to explore how screen time could become more physical, social, and imaginative.

Second, learning opportunities in early childhood are often limited by the materials, space, and specialist knowledge available in each classroom. AI can help educators expand what is possible by supporting the design of new activities, content, and interaction patterns. It should not replace the teacher or speak directly to young children. Instead, it can work behind the scenes, helping educators transform learning goals into age-appropriate experiences.

Finger Wand turns a simple camera and screen into an embodied learning playground. Children use their fingers and body movements to draw, explore letters, create music, and interact with playful visual content—without needing to read instructions, hold a device, or talk to an AI.

What it does

Finger Wand turns an ordinary camera and screen into an embodied learning playground for young children. Using real-time hand tracking, children can control on-screen activities through simple gestures—without touching a device, holding a controller, or reading instructions.

Children can use their fingers like a “magic wand” to:

draw and create pictures in the air trace and explore letters and shapes play with virtual instruments and musical sounds control visual elements through pointing, moving, and pinching gestures receive immediate sound, animation, and success feedback

The experience is designed for classroom use. One child can interact in front of the camera while classmates watch, respond, and take turns, transforming an individual screen activity into a shared learning experience.

Finger Wand uses short visual demonstrations rather than text-heavy tutorials, making each activity intuitive for early learners. GPT-5.6 supports the design of age-appropriate teaching content and interaction logic behind the scenes, while children engage naturally through movement, music, visual feedback, and play.

Only a standard camera and a screen are required—no specialized sensors, headsets, or wearable devices.

How I built it

Finger Wand was built through an iterative collaboration between my teaching experience, ChatGPT, and Codex.

For this age group, I believe AI is better suited to working behind the scenes—helping educators design learning content, select age-appropriate activities, and translate teaching goals into intuitive interaction patterns—rather than speaking directly with young children.

I used ChatGPT as a creative and pedagogical thinking partner. Through prompting and discussion, I explored activity ideas, refined learning objectives, considered children’s developmental needs, and designed interactions that could be understood through movement and visual feedback instead of written instructions.

I then used Codex to turn those ideas into a working browser-based prototype. Codex supported the implementation and refinement of features such as camera-based hand tracking, gesture controls, drawing interactions, musical feedback, activity states, and the user interface.

The development process followed a simple cycle:

Observe a classroom need → discuss and design the experience with ChatGPT → build and test it with Codex → refine the interaction from an educator’s perspective.

This workflow allowed me, as a teacher without a traditional software-engineering background, to transform practical classroom insights into a functioning interactive learning product.

Challenges we ran into

One of the biggest challenges was translating musical ideas into interactions that feel natural, responsive, and expressive for young children.

1. Creating meaningful musical interactions Simulating musical instruments through hand gestures is more complex than simply triggering individual sounds. The interaction needs to respond smoothly to movement while still helping children understand musical concepts such as pitch, rhythm, dynamics, and timbre.

I also explored the idea of an AI choir that children could conduct through gestures. In the future, children could rehearse and develop their own virtual choir through a gamified experience—controlling tempo, dynamics, entrances, and musical expression. However, creating a musically convincing and responsive simulation remains a significant technical and design challenge.

1. Turning children’s drawings into music Another concept was to let children draw musical notation on paper, capture it with a camera, arrange or edit the notes, and then hear their composition performed by an AI band or orchestra.

This could help children connect visual notation, creative decision-making, and the resulting sound. The current prototype does not yet include the recognition, editing, and music-generation pipeline required to make this experience reliable and accessible for young learners.

1. Building a coherent prototype within a short timeframe Finger Wand went from its initial concept to a working Build Week submission in less than one week. Because of this compressed timeline, I focused on demonstrating the core interaction: using an ordinary camera to track a child’s hand and transform movement into immediate visual and musical feedback.

The current version is therefore an early prototype rather than a complete product. Several activity concepts still need further development, and the UI and UX require more testing and refinement—particularly around gesture accuracy, visual clarity, classroom flow, accessibility, and age-appropriate feedback.

Despite these limitations, the prototype demonstrates the central idea: a standard camera and screen can become an embodied learning environment where children create and learn through movement. The challenges encountered have also helped define the next stage of development.

Accomplishments that I'm proud of

As a teacher without a software-engineering background, I was able to independently turn a classroom idea into a working prototype. With ChatGPT and Codex, I could communicate my ideas, test interactions, request changes, and continuously refine the product through natural-language prompting.

This project showed me that educators can take a more active role in building and improving the tools they want to use in their own classrooms—and participate directly in the future of educational innovation.

What I learned

I spent a great deal of time exploring how young children naturally interact, what captures their attention, and what current technology can realistically support. This project pushed me to think more deeply about designing intuitive, age-appropriate experiences and taught me how to communicate ideas more clearly through prompts so that ChatGPT and Codex could produce results closer to my vision.

What's next for Finger Wand

I hope to connect with educators, designers, and developers who share the same vision for more interactive and embodied learning.

Next, I plan to expand Finger Wand with more age-appropriate, inspiring, and educational activities, while testing the platform with children and teachers to improve its usability, learning value, and classroom experience. I am also interested in opportunities where I can continue exploring and improving technology-supported teaching methods.

Built With

• chatgpt
• codex