Spiccor

Spiccor: upload your sheet music, play, and get note-by-note feedback on pitch, rhythm, vibrato and posture, like a private teacher but always available.

Comment

Inspiration

Most violin students practice alone for six days and see a teacher for one. In between, nobody tells them which note was flat, which beat they rushed, or that their wrist collapsed in measure 12. We wanted to build the teacher who listens to every note: upload your music, play it, and get feedback tied to the exact notes on the page. We named it Spiccor, after spiccato, the bouncing bow stroke.

What it does

  1. Upload your sheet music. A PDF scan or MusicXML. Spiccor reads the notation and flags measures that might need proofreading.
  2. Play. Record live with microphone and camera, or upload a recording or video. While you play, you get a live tuner and a skeleton overlay of your body and hands.
  3. Get note-by-note feedback. Spiccor works out which passage you played (you don't have to play the whole piece, and an orchestral intro or tuning beforehand doesn't confuse it). Then it grades:
    • Notes: right, wrong, missed, plus extra or replayed notes.
    • Pitch: cents sharp or flat, relative to your own tuning.
    • Rhythm: early or late against your own tempo, so rubato is allowed.
    • Speed: your tempo against the written one, graded gently, because practicing slowly is normal.
    • Vibrato, for every note: speed, width, how evenly it follows a smooth wave, when it starts, and whether it widens or narrows, with a plot of your pitch against the fitted sine.
    • Posture: body and hand tracking compared with reference players.
  4. Review on the score. Click any note to hear it played by a sampled solo violin, see what was measured, and read what to practice next.

After you finish recording, you can speak to a coach who will provide feedback based on the grades you got in each category (feel free to turn it off if you'd prefer reading, or simply chat with it through text!). A vision-language model (NVIDIA Nemotron, via Bitdeer) reviews key frames to evaluate posture, and you can ask follow-up questions out loud through Cartesia speech and Moss conversation memory.

How we built it

Spiccor has four independent analysis components that exchange results through files using a shared per-note table, with one row per note in the score. Each component has its own Python environment.

  • Score and alignment. Optical music recognition with homr, plus Audiveris as a second opinion to repair measures that don't add up. A music21 score model and Verovio engraving put feedback on the page. Audio-to-score alignment uses chroma-based dynamic time warping (synctoolbox). A custom note matcher (minimum-cost alignment allowing skips, extras and restarts) compares what you played with the score and finds the passage you played inside a long movement.
  • Pitch, transcription and vibrato. CREPE (PyTorch) and pYIN track pitch. To turn audio into notes, we tried a CREPE note segmenter and Basic Pitch (ported to ONNX Runtime), then moved to MUSC, a violin-specific transcription model (ISMIR 2023), with every note's pitch measured by CREPE. Against reference MIDIs, the F1 score on Hilary Hahn's Sibelius rose from 0.689 to 0.775. Vibrato is fitted a few cycles at a time with

$$c(t) = a + b\,t + A \sin(2\pi f t + \varphi),$$

  • speed: the rate (f);
  • width: the peak-to-trough size (2A);

  • evenness: the share of variance the sine explains ((R^2), after removing tracker noise).

    • Grading and comments. Category scores, weights and comment templates. Every number in a comment must come from the measurements behind it, which keeps the feedback honest.
    • Posture. MediaPipe pose and hand landmarks with OpenCV, live in the browser over WebSockets and offline on recorded video.
    • The studio. A FastAPI backend with SQLite storage runs each analysis component as a persistent worker. The frontend is React + TypeScript + Vite, recording through a Web Audio AudioWorklet. A quick preliminary pass gives feedback in seconds, then the full models refine it in the background.

Challenges we ran into

  • Recordings rarely match the whole score. Our first alignment stretched a one-minute excerpt across a four-minute movement, so everything unplayed counted as missed. Good recordings scored 4/10. We built passage location: score and recording are matched with free ends, consistent runs are chained at a steady tempo, and sparse edges are trimmed. Grades on the same recordings rose to 8–9. Matching each take against every score, all 56 pairings came out right: each take was found in its own piece and rejected by the others.
  • Tuning noise looks like music. Our demo study starts on the open D string, so tuning that string produced exactly the first note's pitch. The matcher pinned the first note seconds early, and a take played to a metronome got 2.5 for rhythm. We made sounds before and after the passage free, and short blips and split notes weak evidence, and it now scores 10.
  • The pitch tracker bends vibrato. CREPE reads pitch from 20-cent bins and pulls readings toward their centers. A true −30 to +30 cent swing on A5 read as −20 to +22, and a 15-cent vibrato on A5 disappeared entirely. We refine each frame with YIN's continuous estimate, accurate to within 2.5 cents from G4 to D7.
  • Grading fairly. Busy pieces were graded worse on pitch because the tracker sometimes measured a different note or a double stop. The rhythm grade also first used a fixed 75 ms window, which punished slow practice and rubato. Both are now relative to what the player is actually doing.
  • Making bow-hand recordings comparable. Reference videos varied in camera angle, distance, and hand visibility, so raw landmark positions were not directly comparable. We converted landmarks into joint angles, normalized tracking disagreements by palm size, and filtered unreliable frames before building the reference bank. We also required agreement across multiple performers so one recording could not define the standard. Camera perspective remains a limitation, so the system can withhold feedback when the hand is not clear enough to assess.

Accomplishments that we're proud of

  • An end-to-end studio: a scanned PDF and a phone recording go in, and feedback anchored to each note comes out.
  • Feedback that holds up when tested: wrong and missed notes we deliberately planted in a test recording were caught exactly, and every error we fixed was found by measuring real takes, not guessed.
  • Vibrato technique as a number that musicians recognize: speed, width and evenness.
  • Four independent components, one studio: we built in parallel around a shared file format, then brought everything together into one app.
  • Faster feedback: we worked hard to reduce evaluation time through persistent workers, caching, and selective video sampling. A quick preliminary pass gives users feedback in seconds while the full analysis runs in the background.

What we learned

  • Always check one layer down. Several "grading" bugs were really problems in pitch tracking, alignment or transcription, and we only found them by plotting raw curves and checking against reference MIDIs.
  • Musical context matters more than raw accuracy. Rubato, practice tempo, ornaments and the open string you just tuned all have to count as normal playing, not errors.
  • Transcription needs domain-specific models. A violin-specific model beat general-purpose ones clearly on real performances.
  • Reliable posture feedback starts with reliable measurements. Camera angle, distance and hidden fingers can change what a hand appears to be doing. We learned to compare joint angles, check agreement between landmark models, and withhold feedback when the video does not provide enough evidence.

What's next for Spiccor

  • Musicality advice: pass each phrase's vibrato and dynamics to a language model that knows the piece, its era and character, for advice on interpretation as well as technique.
  • Better double stops and fast passages: stronger multi-note transcription, and intonation judged by intervals.
  • Calibration: tune thresholds against recordings of students and professionals, and let teachers review and annotate.
  • Practice loops: loop a flagged passage at a slower tempo, with live score following while you play.

Built With

  • audiveris
  • basic-pitch
  • bitdeer
  • cartesia
  • crepe
  • cursor
  • fastapi
  • homr
  • librosa
  • mediapipe
  • moss
  • musc
  • music21
  • nvidia-nemotron
  • onnx-runtime
  • python
  • pytorch
  • react
  • sqlite
  • synctoolbox
  • typescript
  • verovio
  • vite
Share this project:

Updates

Submission history