OMR Sheet Music Viewer

Inspiration

I've recently restarted to practice the piano, but I've always been terrible at sight reading scores. I tend to memorize music easily, and my music reading abilities have never developed.

Most music score viewers are just static PDF viewers with support for page turners, maybe some annotation, and nothing else. I needed something that helped me understand the scores quicker, so I could play more pieces. Ideally it should be a single tool that opens a PDF, extracts all the notes from it and presents it as an overlay on top of the original PDF in a way that I can navigate it. Such a tool didn't exist, and I decided to make it myself.

What it does

The app transforms static sheet-music PDFs into interactive, playable scores. It uses optical music recognition to identify notes on each page, then overlays that information directly onto the original music so notes and chords can be selected, highlighted, and followed during playback.

The app also predicts piano fingering for both hands, provides realistic piano audio, and supports customizable keyboard shortcuts and MIDI controls for hands-free navigation. Recognized scores can be exported as MusicXML for editing in compatible notation software. Processing happens locally on the user’s device, keeping their sheet music private.

How we built it

We built OMR Sheet Music Viewer as a three-part local desktop application. The interface uses React and TypeScript, packaged as a lightweight desktop app with Tauri. A Rust layer connects the interface to the operating system: it handles file selection, secure access to cached files, worker-process management, system keyboard settings, and MIDI input.

When a PDF is opened, a Python worker renders each page at high resolution and sends an optimized copy through HOMR, an open source project that performs optical music recognition. We had to fork HOMR because its standard MusicXML output did not contain the visual data needed to reliably position recognized notes over the original PDF. The fork produces an additional visual sidecar containing note coordinates, contours, bounding boxes, and links to the corresponding MusicXML notes.

The app maps that geometry back onto the high-resolution page, allowing the React interface to draw accurate interactive overlays. Notes and chords can then be selected, highlighted, and followed during playback. Results from every page are combined into a single MusicXML score.

To improve performance, PDFs are identified by a cryptographic hash, and completed pages and recognition results are cached. Music playback uses Tone.js with sampled piano audio, while local ONNX models predict fingering for both hands. The Rust backend also listens for MIDI input, allowing pedals and controllers to operate the viewer. The entire pipeline runs locally, keeping the user’s sheet music private.

We used GPT 5.6 Sol xHigh heavily in this project.

Challenges we ran into

The biggest challenge was not simply recognizing the music - it was preserving the connection between each recognized note and its exact position on the original PDF. Standard HOMR output provided the musical structure as MusicXML but not enough pixel-level information for accurate visual overlays. Therefore we had to fork HOMR to fill that gap.

Coordinate accuracy introduced another challenge. The app displays each page as a high-resolution 300-DPI image, while HOMR processes a smaller image optimized for recognition. We had to account for resizing and preprocessing, then translate the recognition geometry back into the display image’s coordinate system so highlights remain aligned with the printed notes.

HOMR also produces a separate score for each page. Combining those files into one usable document required validating parts and staves, renumber measures, preserve page breaks, and prevent identifiers from colliding.

Recognition can be computationally expensive, so responsiveness was another concern. We moved PDF rendering and HOMR processing into a separate Python worker controlled by the Tauri backend. A structured event protocol reports page progress, failures, and diagnostic logs to the interface without freezing it. We also created a version-aware cache so completed pages can be reused while outdated results are invalidated safely.

Linking playback and fingering back to the visual score presented a similar data-matching problem. Notes from the MusicXML, visual sidecar, playback timeline, and fingering models all needed stable identifiers. Maintaining those relationships lets the app highlight the correct note, play its pitch, display the predicted finger, and write that fingering back into MusicXML.

Finally, hardware support brought platform-specific challenges. MIDI drivers can behave differently across operating systems—and may sometimes stop responding entirely—so MIDI discovery runs outside the interface thread with a timeout. This ensures that a faulty device or system MIDI service cannot prevent the rest of the viewer from working.

Accomplishments that we're proud of

What we learned

We learned that successful optical music recognition requires more than producing correct MusicXML. For an interactive viewer, every musical symbol must remain connected to its visual location on the source page. Stable identifiers and detailed geometry are just as important as the recognized pitches and rhythms.

We also learned that a single image cannot serve every purpose equally well. High-resolution pages are best for reading, while consistently resized images perform better for recognition. Keeping separate display and recognition images—and carefully translating coordinates between them—gave us better visual quality without compromising the OMR pipeline.

Another lesson was to design around imperfect recognition. Real scores may contain blank pages, unusual layouts, scanning artifacts, or notation the model cannot interpret. Processing each page independently, preserving partial results, and allowing targeted retries made the application far more resilient.

Building a local AI application also proved to be as much a systems challenge as a machine-learning challenge. Process supervision, caching, progress events, validation, logging, and graceful recovery were essential for turning HOMR into a dependable desktop experience.

We found that open formats still require careful engineering. MusicXML gave us a strong foundation, but merging page-level scores, maintaining unique identifiers, preserving page breaks, and adding fingering data all required additional processing.

Finally, we learned to isolate optional hardware features from the core experience. MIDI devices and drivers can be unpredictable, so timeouts and failure boundaries are necessary. A MIDI problem should disable MIDI—not freeze the entire application.

What's next for OMR Sheet Music Viewer

OMR Sheet Music Viewer is still a proof of concept. Its main goal was to demonstrate that recognized notation could be positioned accurately over a PDF and used to create an interactive experience with highlighting, playback, fingering, and MIDI controls. The next phase is about turning that technical foundation into a dependable score viewer.

Recognition quality is the largest challenge. HOMR can identify a substantial amount of musical content, but the results are not yet consistently accurate—particularly for rhythm. Incorrect durations, voices, rests, ties, or measure structure can make real-time playback unreliable even when the detected pitches look correct. Improving the recognition pipeline, exposing confidence information, and giving users a way to correct mistakes will be important next steps.

Fingering prediction also needs more work. The current hand assignment assumes bass-clef notes belong to the left hand and treble-clef notes belong to the right, which does not handle cross-staff notation, hand crossings, shared voices, or other advanced piano writing. The fingering model also has edge cases involving chords, repeated notes, large jumps, and unusual passages. A better model— along with editable fingerings and more musical context—will likely be required.

The application has focused primarily on interactivity, so it is still missing many features expected from a mature score viewer. These include stronger document and page navigation, library and recent-file management, additional viewing layouts, bookmarks and annotations, printing and export options, richer playback controls, and more complete accessibility support.

Other important next steps include:

• Supporting manual correction of recognized notes, rhythms, and fingering.
• Testing against a much broader range of scores, including scans, complex piano music, orchestral parts, and non-standard layouts.
• Improving processing speed, model startup time, and memory usage.
• Packaging the Python runtime, HOMR fork, and model files into straightforward cross-platform installers.
• Making page merging more robust when instrumentation or staff structure changes between pages.
• Expanding beyond PDF input to image files and existing MusicXML scores.
• Improving recovery when recognition fails and making partial results easier to understand.
• Refining real-time playback with looping, count-in, metronome, seeking, and practice-focused controls.
• Continuing to develop the HOMR visual sidecar format—and potentially contributing the approach upstream—so recognized musical data and source-page geometry remain reliably connected.

The proof of concept shows that local OMR can power a genuinely interactive sheet-music experience. The next challenge is improving musical accuracy and building the everyday viewing, editing, and practice features needed to make it useful beyond the prototype stage.

Built With

• javascript
• python
• react
• rust
• tauri
• vite