Sing Coach

• 

  Landing page

• 

  Inference Output and Pitch Chart

Inspiration

Existing singing apps do shallow on-device scoring: a pitch number against a target melody. The deeper coaching experience — analysing vibrato and vocal cracks, then explaining them in natural language — is the kind of thing teams reach for a GPU or a hosted LLM API to build. We wanted to find out whether the whole pipeline fits on a cheap CPU instance, and to measure it honestly rather than assume and then we made SingCoach

What it does

Record a few seconds of singing in the browser. SingCoach tracks pitch every 10 ms, segments the take into notes, and measures for each one how far off the ideal pitch it was (in cents), how much it wobbled, and whether vibrato is present. It flags voice cracks, pitch drift, and inconsistent loudness. A quantized Llama 3.2 3B then reads those measurements and writes specific coaching advice.

It deliberately does not claim to judge whether you are a good singer — reference-free singing quality assessment is an open research problem. It measures well-defined acoustic properties and coaches you on those.

How we built it

• Pitch: librosa pYIN (classical DSP) with torchcrepe tiny/full as neural comparison backends, all on a common 10 ms hop so they are interchangeable.
• Metrics: numpy — note segmentation, cents deviation, stability, drift, vibrato rate/extent, crack detection, LUFS-style loudness.
• Coach: llama.cpp built with GGML_CPU_KLEIDIAI=ON, serving Llama 3.2 3B Instruct Q4_0. The model never sees audio, only a compact JSON summary — that is what keeps a 3B model grounded and the prompt small enough to be fast.
• API: FastAPI, with an SSE endpoint so coaching streams token by token.
• Frontend: Next.js static export served directly by caddy; no Node process in production.
• Deploy: AWS Graviton (t4g.medium), systemd units, HTTPS via caddy.

Challenges we ran into

• A first cross-architecture benchmark showed Arm 9× faster at pitch tracking. It was false: the two machines had different librosa versions (1.0 sped up pYIN ~16×). We pinned the version and re-measured rather than publish it.
• Graviton2 (Neoverse N1) lacks the i8mm instructions that KleidiAI's quantized kernels rely on, which shapes what the KleidiAI comparison can show.
• A 3B model at Q4 will confidently invent metrics and give physically impossible vocal advice unless the prompt forbids it explicitly.

Accomplishments that we're proud of

1. Built and deployed a working AI singing coach that runs its entire pipeline — pitch analysis plus a 4-bit Llama 3.2 3B writing the feedback — on a single Arm CPU instance costing three cents an hour, with no GPU anywhere.

2. Found that enabling KleidiAI is not a universal win

3. Established, with reproducible benchmarks across Graviton2, Graviton4 and Intel Sapphire Rapids on identical software, that Arm generates text 2.4× faster and 2.7× cheaper per token

What we learned

CPU-only inference is entirely viable for this class of application, but the engineering that matters is not "make the model faster" — it is deciding what actually needs a model at all. Pitch tracking, note segmentation, and defect detection are cheap deterministic code; only the natural-language explanation needs an LLM, and feeding it numbers instead of audio is what makes a small quantized model trustworthy.

What's next for Sing Coach

• Melody-reference mode (compare against a MIDI target for karaoke scoring)
• Per-note coaching timeline rather than one paragraph
• Longer takes with phrase-level breakdown

Built With

• arm
• aws-graviton
• caddy
• librosa
• llama
• llama.cpp
• python
• torchcrepe