MagicChan

You draw the spell. StackChan brings it to life.

Comment

Inspiration

Most gesture demos stop at recognition on a screen. I wanted magic to feel like a shared act: the player provides intent with a wand, and a small robot gives that motion a face, light, and voice.

What it does

MagicChan recognizes four wand gestures—glow, dim, orbit, and burst—using the camera in an M5Stack CoreS3. When a spell is recognized, StackChan completes it with an expression, twelve LEDs, and a custom voice. The final physical path runs locally on the device without a Mac or cloud API.

How I built it

The illuminated wand tip is tracked as a calibrated color marker. A C++ stroke segmenter and gesture recognizer run inside the CoreS3 firmware, then dispatch directly to MagicChan's reactions. The four short voice assets were designed and generated locally with Irodori-TTS and embedded in flash for non-blocking playback. I also kept a TypeScript browser implementation as a tuning surface and made both recognizers consume the same checked-in golden fixtures. The removable magic hat and wand LED marker were iterated as 3D-printed hardware.

How I used Codex and GPT-5.6

Codex and GPT-5.6 helped port the recognizer from TypeScript to C++, build shared golden fixtures, diagnose recorded gesture traces, integrate the four local voice assets, and generate and validate the 3D-print files. I made the product and experience decisions: the two-participant concept, the four gestures and reactions, the voice, physical fit, speaker level, and the deliberate gesture speed used in the demo.

Challenges

Real wand motion was less clean than synthetic input. An upward flick could leave the camera frame and the return motion could be misread as a second downward spell. Recorded traces showed that tracking loss was useful stroke-ending evidence, and a shared cooldown fixture prevented the duplicate event. Audio also had to play without blocking the roughly 20 fps tracking loop. Finally, the product-version StackChan enclosure required a new removable hat mount rather than the community-version attachment.

Accomplishments

  • Four spells run locally from camera tracking through recognition and physical reaction.
  • TypeScript and C++ pass the same 13-case gesture fixture without special hardware.
  • The four embedded voice assets play without stopping the tracking loop.
  • The removable MagicChan hat and thin adapter mount fit the physical StackChan.

What I learned

Physical interaction is shaped as much by timing and feedback as by classifier accuracy. A gesture can be intentionally slower if the character makes that pace feel like part of casting a spell. I also learned that a robot becomes more engaging when it contributes to the action instead of merely confirming a command.

What's next

I would add clearer calibration feedback, teachable gesture pacing, and more environmental reactions. A future version could connect successful spells to local smart-home actions while keeping recognition and the core character experience on-device.

Built With

Share this project:

Updates