Mumu

What if your story could come to life? Meet Mumu, an AI-powered digital reader that transforms flat stories into an immersive, interactive and premium reading experience.

Comment

Inspiration

Mumu was inspired by the growing popularity of manga and manhwa and the idea that, while comics are incredibly immersive visually, they are still usually a silent experience. Readers imagine the voices, atmosphere, and emotional tone in their heads, but I wanted to explore what it would feel like if that experience came alive more like anime while still preserving the feeling of reading.

The hackathon theme, “digital sidekick,” gave the idea a stronger identity. I wanted to build a reading companion, Mumu the orange cat, who can join you throughout your reading experience. Mumu is a cat sidekick that adds charm, personality, and utility to the product.

What it does

Users can import their own comic chapters, which are analyzed using Google Gemini to identify dialogue, recognize characters, understand emotions, and determine the atmosphere of each scene.

As readers scroll, Mumu enhances the experience through:

  • AI character voices: Characters receive distinct voices that remain consistent between chapters, making dialogue feel more alive.
  • Dynamic music and sound effects: Audio changes to match the atmosphere of each scene, from peaceful conversations to intense battles.
  • Scroll-synchronized audio: Dialogue plays as the reader progresses through the story.
  • Interactive cat companion: Mumu walks around the screen, reacts to different scenes, and responds when you pet or drag him.
  • Reading reminders: Mumu tracks your reading time and starts meowing when you've been reading for too long, encouraging you to take a break.
  • Chapter recaps: Mumu helps you catch up on previous chapters by narrating a summary of the story.
  • Offline reading: Once a chapter is processed, readers can enjoy their comics and generated audio without an internet connection.

The goal isn't to replace the original manga experience, but to enhance it by making it feel like you're sharing the adventure with a companion.

How I built it

I built Mumu as a pixel-art progressive web application using React, TypeScript, Vite, and Tailwind CSS, with an Express.js and Node.js backend.

At the center of the application is Google Gemini, which powers the comic understanding and audio generation pipeline.

When a user uploads a comic chapter, the backend divides the pages into smaller segments for analysis. Gemini identifies speech bubbles, extracts dialogue, determines which characters are speaking, and analyzes the emotional tone of the scenes.

I then use Gemini's text-to-speech capabilities to generate character voices, assigning consistent voice profiles so characters sound recognizable across chapters.

On the frontend, I built a custom audio playback system using Howler.js that synchronizes dialogue, background music, and sound effects with the user's scrolling position. I also used Dexie.js and IndexedDB to store processed chapters and audio locally for offline access.

For Mumu himself, I created a pixel-art animation system driven by a state machine. This controls his movement, interactions, and reactions, allowing him to walk, sit, climb, get picked up, and respond to different story moods. His behavior is connected to the reader so that changes in the story can influence how he acts.

As a solo developer working within a one-day hackathon, I also used Claude Code to help coordinate development across the backend, frontend, and Mumu's character system.

Challenges I ran into

One of my biggest challenges was getting AI to understand manga beyond simply recognizing text. Comic layouts can be unpredictable, with speech bubbles positioned in different locations and characters speaking across multiple panels. Determining who is speaking and understanding the context of a scene required more than basic text extraction.

Another challenge was making Mumu feel alive rather than like a static decoration. I wanted his movements and reactions to feel natural without distracting readers or interfering with scrolling. This meant carefully designing his animations, interactions, and behavior states.

Synchronizing audio with scrolling was also difficult because everyone reads at a different pace. I needed to account for readers scrolling quickly, pausing, or moving backward without causing overlapping dialogue or repetitive audio.

Finally, balancing all these systems within a single day was challenging. I had to prioritize the core reading experience while making sure Mumu's personality and interactions remained central to the application.

Accomplishments that I'm proud of

My biggest accomplishment is bringing together AI-powered storytelling and an interactive digital sidekick into one experience.

I'm especially proud of Mumu's personality. Rather than creating a generic assistant that only answers questions, I wanted to make a companion that communicates through animations, reactions, and playful interactions.

I'm also proud of:

  • Designing a system that understands comic dialogue, characters, and scene emotions using multimodal AI.
  • Connecting the story's emotional atmosphere to both the audio experience and Mumu's reactions.
  • Creating an interactive pixel-art cat that users can pet, drag, and interact with while reading.
  • Building an offline-first reading experience that doesn't require a constant internet connection after processing.
  • Tackling a project involving AI, audio processing, animation, and frontend development as a solo hackathon participant.

What makes Mumu special to me is that the AI isn't only used to generate content. It also helps give a little digital character the ability to respond to the story you're experiencing.

What I learned

Through this project, I learned how multimodal AI can interpret visual storytelling and turn that understanding into something interactive.

Working with Gemini taught me how to combine image analysis, structured data extraction, contextual reasoning, and text-to-speech generation in a single pipeline.

I also learned a lot about building interactive characters. Creating Mumu introduced challenges involving animation states, user interactions, and connecting AI-generated scene information to real-time character behavior.

Another major takeaway was the importance of modular architecture, especially when building under time constraints. Separating the AI pipeline, audio system, offline storage, and Mumu's behavior made it easier to develop and integrate different features.

Most importantly, I learned that a digital sidekick doesn't have to be a chatbot. It can communicate through actions, expressions, and small interactions that make an otherwise ordinary experience more engaging.

What's next for Mumu

There are plenty of ways I'd love to expand Mumu beyond the hackathon.

One of my biggest goals is to make Mumu even more expressive. I'd like him to develop a wider range of reactions, recognize more specific story events, and respond differently depending on what's happening in the manga.

I'd also like to explore:

  • More expressive character voices: Greater customization of voices, emotions, and delivery styles.
  • Smarter scene understanding: Better recognition of character interactions, unexpected plot twists, and changes in atmosphere.
  • More Mumu interactions: New animations, activities, and ways for readers to play with their companion.
  • Personalized reading experiences: Allowing Mumu to remember reading progress, favorite series, and user preferences.
  • Improved audio synchronization: More accurate panel-level dialogue timing and richer sound effects.
  • Expanded offline support: Making it easier to import, manage, and revisit entire manga libraries.

Ultimately, I want Mumu to become a companion readers look forward to seeing whenever they open a new chapter.

Because sometimes, the best part of experiencing a great story is having someone to experience it with.

Built With

Share this project:

Updates

Submission history