Inspiration
Inspired by the pixel-art theme, we immediately thought of retro platformers—games where movement and color drive the adventure. We wanted to flip that idea: what if the world lost its color? That’s how Chromaticity was born: a motion-controlled platformer where players use real movement to restore a fading world. One thing that makes this project truly unique is the Toronto-man accent used in the game’s narration. It’s a voice and vibe that personally resonates with us, adds humor, and gives the game a distinct identity that players instantly recognize.
What it does
Chromaticity blends physical movement with pixel-art gameplay. Players jump, run, and fire using: -Camera pose detection (jump / stand) -A custom ESP32 motion controller with IMU + button -A live LCD showing inventory -Narrative voice-lines in a Toronto-man accent
Players collect three crystals, restore color to the world, and overcome obstacles and a final boss—all through motion-based controls. Chromaticity is designed to be fun for families, classrooms, and casual groups. Because the game uses physical movement, it naturally encourages activity, laughter, and group participation.
How we built it
We divided the project into hardware and software tracks. Juan built the physical controller, integrating an ESP32, MPU6050 IMU, 16×2 LCD, and tactile buttons. With support from AI assistants, we implemented community-standard sensor-fusion algorithms to transform raw accelerometer and gyroscope data into smooth, reliable gameplay signals. The controller sends JSON packets over BLE carrying movement speed, orientation, and button events. John built the web-based platformer, using Phaser + Three.js for movement, physics, and visual rendering. For motion input, we used TensorFlow MoveNet for camera-based jump detection and fused that with IMU data through an Input Fusion Layer that unifies pose, IMU, and button events into one control stream. Zustand manages real-time global game state across subsystems. We used Snowflake to store calorie data and gameplay metrics, while the Gemini API analyzes that history and generates personalized Toronto-man style voice lines. ElevenLabs then converts those lines into dynamic in-game audio.
Challenges we ran into
One of our biggest challenges was understanding the physics behind the IMU. We had to iterate repeatedly to achieve reliable arm-swing detection, using serial plotting tools to study movement patterns and tune our algorithms. Keeping pose tracking (MoveNet) and IMU inputs perfectly in sync required careful calibration and constant playtesting—too sensitive and players moved by accident, too strict and they had to exhaust themselves. On the hardware side, we dealt with LCD corruption during BLE transmission, which we solved by restructuring RTOS tasks. Stabilizing ESP32 communication and ensuring clean, consistent JSON packets was also a major hurdle, especially under rapid movement. From a design perspective, making a greyscale world readable and visually appealing took multiple lighting and contrast passes. We also spent time refining the Toronto-man narration so it felt natural, funny, and not overly long. And throughout development, we had to carefully balance physical gameplay difficulty with accessibility—ensuring the game feels active and immersive without overwhelming players.
Accomplishments that we're proud of
We built a fully functional motion controller from commodity hardware that feels responsive and natural in gameplay. Our energy-based movement detection with hysteresis thresholds produced smooth, reliable controls, and we successfully integrated BLE communication, IMU processing, LCD animations, and button inputs into one cohesive device. The animated LCD icons and low-ammo warnings give the hardware its own personality. On the software side, we implemented camera vision modelling for real-time body movement and fused it with our IMU signals for consistent, full-body interaction. We also crafted a unique Toronto-man narrator, giving the game a cultural identity that sets it apart. Finally, we integrated Snowflake + Gemini + ElevenLabs into a seamless pipeline for analytics and dynamic voice lines, and built an experience that’s fun and accessible for families, classrooms, and casual players.
What we learned
We learned how important sensor physics is when building real hardware, and how to merge embedded systems with a browser-based game engine. We gained experience in wireless communication, sensor fusion, real-time feedback loops, and UI constraints of small LCD devices. We also learned how AI tools can accelerate development—helping us prototype, debug, and focus more on system design than boilerplate code. Most importantly, we learned how to tie physical movement into digital narrative in a way that feels meaningful.
What's next for Chromaticity
Multiplayer mode where players collaborate, race, or battle to restore color Crouch detection to expand the movement vocabulary and obstacle interactions More handcrafted levels with new mechanics, color-restoration sequences, and boss encounters Online leaderboards for calories burned, speedruns, accuracy, and multiplayer rankings Improved motion algorithms with better IMU filtering, smoother fusion with MoveNet, and more reliable gesture detection Expanded controller features, including haptic feedback, RGB indicators, better LCD animations, and additional programmable inputs for new in-game actions Our long-term goal is to evolve Chromaticity into a polished, replayable experience that blends fitness and fun—perfect for families, classrooms, and casual players.
Log in or sign up for Devpost to join the conversation.