Melody Rain

Notes fall like rain. Music comes alive.

Comment

Inspiration

I studied piano for many years as a child. As school, work, and adult life became busier, I gradually stopped practicing. But my love for music never disappeared. I still enjoy looking at sheet music and imagining what it would feel like to learn a piece and play it with my own hands.

Most MIDI sheet-music visualizations are useful for previewing a piece or following its progress, but they often feel more like functional timelines than living performances. I wanted something that could express the movement, color, and joy I still feel when I look at a score.

Music feels like rain to me. It returns to our lives from time to time and quietly nourishes our emotions and memories, even when we are too busy to stop and truly experience it. Melody Rain is a small doorway back into that feeling. Even for people who no longer have time to touch an instrument, a short video can let them see notes emerge from a score, fall with the music, and briefly reconnect with the beauty of music.

What it does

Melody Rain transforms traditional sheet music into an animated, portrait-format performance. It combines MusicXML or MXL notation, MIDI timing, and MP3 audio on one synchronized timeline. As the piece plays, notes, chords, rests, beams, and related notation fall toward the score, react when they land, and reveal color or imagery through visual masks.

The score follows the performance with a three-stage camera: it remains still at the beginning, scrolls at a constant speed once the music reaches the visual midpoint, and stops near the end so the final notes can complete their fall. Users can customize the background, colors, transparency, animation style, score layout, title, and playback speed. The interface is bilingual in English and Chinese.

Melody Rain can also export the complete performance as a 9:16 MP4 video in standard or high quality, making the result suitable for sharing in the short-video formats where many people now encounter music.

How we built it

Melody Rain is a local-first React and TypeScript application built with Vite and SCSS. OpenSheetMusicDisplay renders MusicXML as SVG sheet music, while @tonejs/midi provides MIDI timing. A shared transport keeps audio playback, score movement, and note animation synchronized.

The animation system maps MIDI events back to visible score elements and computes every falling, landing, and resting state from an absolute timeline. This same deterministic state model powers both interactive preview and video export.

For export, a local Express service launches an installed Chrome or Edge browser through Playwright. It renders the performance frame by frame and pipes PNG frames to FFmpeg, which creates a 30 FPS H.264/AAC MP4 with the source audio. Standard export produces 540 × 960 video, while high-quality export produces 1080 × 1920 video.

Codex was used as an active development collaborator throughout the project: translating visual and emotional ideas into implementation plans, tracing rendering and synchronization issues across the codebase, refactoring the UI and SCSS structure, building the deterministic export pipeline, writing tests, and keeping the technical specification and README aligned with the product.

GPT-5.6, accessed through Codex, was the primary AI model used throughout the creation of Melody Rain. It helped turn the initial emotional concept into the product and technical specification, draft and maintain the README, design the architecture, generate and refactor the React, TypeScript, SCSS, Express, Playwright, and FFmpeg code, write tests, diagnose bugs, and iterate on synchronization, animation, UI, and export behavior. GPT-5.6 was used as a development collaborator across the project rather than as a model embedded in Melody Rain at runtime; the application itself does not currently call the OpenAI API while a user previews or exports a performance.

Challenges we ran into

The hardest problem was keeping three different representations of the same music aligned: notation from MusicXML, event timing from MIDI, and recorded sound from MP3. A visually convincing result depends on all three agreeing closely enough that a note lands exactly when it is heard.

Another challenge was making preview and export tell the same visual story. Real-time browser playback depends on the clock, while reliable video generation needs every frame to be reproducible at an exact timestamp. We moved the animation and camera behavior toward deterministic calculations based on absolute musical time so the exported video matches the preview.

Video export also exposed browser security and performance constraints. Canvas and VideoFrame APIs can reject images from tainted sources, and rendering browser screenshots frame by frame is much slower than real time. We chose a local, privacy-friendly pipeline using an installed browser and FFmpeg, added standard and high-quality modes, and made long exports visible and cancellable.

Finally, making notation feel expressive without making it unreadable required many small visual decisions: when scrolling should begin, how motion should end, how beamed notes should behave, how much of the background should show through, and how the final system should retain breathing room at the bottom of the frame.

Accomplishments that we're proud of

We are proud that Melody Rain is more than an animation placed on top of a score. The falling elements originate from the notation itself and remain connected to musical timing, score layout, scrolling, and audio playback.

We also built a complete local video workflow. A user can load their own score, MIDI, audio, and background assets; shape the visual identity of the performance; preview it; and export a shareable portrait MP4 without uploading private music files to a cloud service.

Most importantly, the project now communicates the original feeling behind it: sheet music is not a dry progress display. It can move, breathe, and invite someone back into music.

What we learned

We learned that musical visualization is as much about emotional pacing as technical synchronization. A camera that moves too early makes the viewer feel pushed; one that accelerates unpredictably distracts from the score. Holding the page still, beginning a constant movement at the visual midpoint, and allowing the final notes to fall after scrolling stops created a much calmer and more musical experience.

We also learned the value of a single source of truth for time. Sharing transport, camera, and animation-state calculations between preview and export reduced inconsistencies and made complex visual behavior testable.

Working with Codex showed us how an AI coding collaborator can help preserve intent across product design, implementation, debugging, testing, refactoring, and documentation—not only generate isolated pieces of code.

What's next for Melody Rain

Next, we want to make each falling note feel even more alive through richer landing, particle, light, and motion effects while preserving the readability of the score. We also want to improve MusicXML-to-MIDI alignment, add more aspect ratios and visual themes, and make video export substantially faster.

In the longer term, Melody Rain could make a wider range of scores easy to experience and share. The goal is not to replace learning or performing an instrument. It is to offer a small emotional opening: a way for people who have drifted away from music to see it again, feel it again, and perhaps one day return to playing.

Built With

Share this project:

Updates