Moeko-chan!

Moeko-chan!

Inspiration

Everyone uses music players like Spotify 🎧—but who says listening has to be passive? Imagine a dedicated listening companion that truly vibes with you ✨, feels your mood 💭, learns your rhythm 🎶, and grows with your taste. Not just playing tracks, but curating moments—fueling your focus 🚀, lifting your spirits 🌈, and soundtracking the quiet in-between 🌙. Music, but personal. Music, but alive 💫.

What it does

Moeko-chan! is a desktop AI companion that brings your music to life by reacting to audio in real time. It listens to either microphone input or system audio from applications like Spotify and YouTube, analyses musical features such as BPM, energy levels, genre, and mood, and translates those signals into expressive animations. The on-screen chibi character bounces, dances, changes emotions, and even falls asleep when the music goes quiet, creating a playful yet ambient presence on your desktop. With support for transparent mode, always-on-top behaviour, and mood locking, Moeko-chan! is designed to blend naturally into your workflow as a non-intrusive music-reactive overlay.

How we built it

We built Moeko-chan! using Next.js as the core framework and Electron to package it as a cross-platform desktop application. Real-time audio analysis is handled through the Web Audio API, where we extract frequency bands, energy levels, and timing information from live audio streams. For BPM detection, we combined custom beat detection logic with the analyser library as a fallback to improve robustness. Genre and mood classification are implemented using heuristic-based ML scoring that smooths results over time to prevent flickering. These audio signals are then mapped into a sprite-based animation system with physics-inspired motion, while Tailwind CSS and Radix UI enabled rapid iteration on UI components and controls.

Microphone Mode (Default)

Icon: 🎤 Mic
Works out of the box with no special permissions
Captures ambient sound through your microphone
Great for reacting to music playing through speakers
Works on all platforms

Features

Dual Audio Modes - Switch between microphone and system audio capture
BPM Detection - Detects beats per minute (50-200 BPM range) using beat detection algorithms
Mood Detection - Classifies audio mood as chill, energetic, happy or sleep
Animated Character - Bouncy sprite with physics-based animations that react to the beat
Desktop Widget - Floating, always-on-top transparent window (300x350px)
Global Shortcut - Toggle visibility with Ctrl+Shift+V (or Cmd+Shift+V on Mac)
Transparent Mode - Press Ctrl+Shift+T for a fully transparent background showing only the character
Mood Lock - Lock the character to a specific mood animation or an auto mode to automatically adjust the chibi's mood, via the dropdown menu
Custom Sprites - You can use your own custom Moeko-chan!

Challenges we ran into

One of the most difficult challenges was achieving accurate and stable BPM detection in real-world conditions. Early implementations unintentionally capped bass energy values, which completely broke beat detection and caused BPM to freeze at incorrect values. We also encountered false beat detection during startup noise, as well as runtime errors when connecting third-party BPM processors. Beyond audio, synchronising detected moods with sprites and glow effects proved tricky due to stale state and timing issues, requiring a careful refactor of how “effective mood” was computed and propagated across the app. Ensuring smooth transitions during silence, inactivity, and mode switching added another layer of complexity.

Accomplishments that we're proud of

We’re especially proud of building a reliable, end-to-end real-time audio pipeline that handles noisy input, silence, and edge cases gracefully. Features like automatic sleep mode after inactivity, gradual BPM decay instead of abrupt resets, and a fully transparent overlay mode significantly improved the user experience and polish of the app. Fixing critical audio bugs and implementing detailed debug tooling allowed us to confidently tune thresholds, smoothing, and animations. Despite the technical complexity under the hood, the final result feels responsive, expressive, and fun to use.

What we learned

This project taught us that real-time audio processing is extremely sensitive to small implementation details, particularly around normalisation, thresholds, and temporal smoothing. Having strong debugging and visualisation tools is indeed essential when working with audio-driven behaviour and animations. We also learned the importance of balancing responsiveness with stability, as instant reactions can feel jittery, while too much smoothing makes the character feel disconnected from the music. Most importantly, we gained experience designing systems where multiple real-time signals must stay visually and logically in sync.

What's next for Moeko-chan!

Looking ahead, we want to improve the intelligence and expressiveness of Moeko-chan! by experimenting with more advanced machine learning models for mood and genre detection, potentially replacing heuristic scoring with lightweight neural networks. We also plan to expand customisation options, including more character animations, skins, and user-created sprite packs. On the platform side, we aim to improve cross-platform audio capture reliability and explore extensibility through plugins or APIs, turning Moeko-chan! into a flexible foundation for personalised desktop companions.