OmniRiff

OmniRiff is a guitar teacher that lives in your signal chain: it hears and sees you play, finds why you stumbled, and sets up the drill, without you ever putting the guitar down.

Comment

Inspiration

Practicing alone, you can hear that something went wrong without knowing what to fix. Was the chord change late? Did your strumming hand stop moving? Should you slow down, isolate the transition, or try a different exercise?

We wanted a practice partner that could hear a take, use hand movement as additional evidence, and turn the stumble into one specific thing to practice next. Just as importantly, it should keep the player in the flow: no putting down the guitar to navigate another app.

That became OmniRiff, and its little pink coach, Riff.

What it does

OmniRiff combines an instrument, a looper, gesture controls, and an AI practice coach.

You play and record a phrase. Ears analyzes strum timing, gaps, and hesitations. When camera data is available, Eyes adds information about hand movement. Riff uses that evidence to suggest a likely issue, propose a focused drill, and guide another take.

The player approves proposed rig changes before the coach changes sound settings. The system can compare takes and keep a practice journal so a later session has context.

The instrument includes a looper, filter, reverb, bypass, and limiter. Hand gestures and an optional Arduino footswitch provide controls while the guitar stays in your hands.

There are two ways to explore it:

  • In the browser: a playable instrument with a free, clearly labeled simulator, plus optional hosted AI coaching using a privately shared team passcode.
  • On a local rig: a Python audio engine connected to a guitar, audio interface, webcam, and pedal.

Audio processing stays separate from the coach. Camera, pedal, dashboard, or coach failures should not interrupt the local audio path; disconnecting the audio interface itself does interrupt physical audio.

How we built it

The local audio engine uses Python, NumPy, SciPy, and sounddevice/PortAudio. It processes streaming audio with a smoothed filter, reverb, looping, crossfaded bypass, and a limiter. Finished loops are saved as takes outside the real-time audio callback.

Ears detects pick attacks and measures timing using high-frequency energy changes and onset analysis. Eyes analyzes hand movement from MediaPipe’s pretrained hand landmarks. We did not train the hand-landmark model ourselves.

The local engine receives controls over UDP, USB serial, and WebSockets. Versioned message contracts, confidence checks, freshness checks, clamping, and smoothing help keep external controls bounded.

Riff uses LangChain deepagents and Anthropic Claude. It has Ears and Eyes tools/subagents, drill skills, a visible plan, filesystem-backed practice memory, and human approval for proposed rig changes.

The browser instrument uses TypeScript, Vite, and Web Audio. Its interface brings together the waveform, effects, hand view, session plan, and Riff. Hosted coaching runs separately: finished takes and relevant hand/state information are sent to the coach server, while instrument audio processing remains in the browser.

An Arduino footswitch provides physical controls over USB serial. We also added browser-based serial support and a separate voice-microphone selection so spoken requests can use the laptop microphone instead of the guitar input.

Our four-person team coordinated human-directed coding agents through a shared GitHub issue board, scoped ownership, pull requests, and automated checks.

Challenges we ran into

Detecting strums on a ringing guitar

Sustained strings and overtones can resemble new attacks, while soft upstrokes can be easy to miss. We moved beyond raw volume and focused on the high-frequency transient produced by the pick.

Making gestures deliberate

A guitarist’s hands are already moving. We had to distinguish intentional controls from strumming, fretting, missing detections, low-confidence frames, and camera jitter. Real-camera testing exposed regressions that synthetic tests alone did not catch.

Keeping audio independent of AI

Coaching can take time or lose connectivity. The instrument cannot wait for it. We separated coaching from audio processing and kept network, file, and model work outside the real-time callback.

Coordinating several agents and interfaces

Shared contracts, folder ownership, and integration reviews mattered as much as generating code. A change that worked in one demo path could still mislead another path unless its instructions and controls stayed aligned.

Accomplishments that we're proud of

We built an integrated practice workflow spanning audio processing, computer vision, an AI coach, a browser interface, and physical controls.

The project includes a playable browser simulator, a local guitar-processing path, take analysis, practice planning, player-approved rig actions, and persistent practice context.

We also built clearly labeled simulation and replay paths so a fallback demonstration does not have to pretend to be a live performance.

Our automated checks exercise message contracts, audio behavior, browser logic, and offline integration. Team camera trials have also exercised intentional gestures and normal playing behavior.

What we learned

A useful coach needs more than a plausible sentence. It needs evidence from the take, a specific suggestion, and another take that can test whether the suggestion helped.

We learned that hand movement and audio timing can provide complementary evidence, but matching events do not automatically establish a cause.

We also learned to treat a musical instrument differently from an ordinary web app: predictable controls, continuity of sound, and a usable manual fallback are central to the experience.

What's next for OmniRiff

We want to test OmniRiff with guitar teachers and students, improve the specificity of its coaching, and evaluate whether its drills help across repeated practice sessions.

Technical next steps include more robust real-guitar onset analysis, better hand tracking across players and camera positions, stronger longitudinal practice memory, and broader device support.

Longer term, we would like to explore a dedicated pedal and richer analysis of chord transitions and articulation.

Verification and current limits

The gallery includes scripted simulator sessions and synthetic analyzer fixtures, labeled accordingly. These demonstrate software behavior, not measured learning gains.

The hosted coach has been exercised with synthetic takes. Full physical-rig validation, measured guitar-to-output latency, sustained hardware dropout results, and coaching quality on real players remain separate checks. We have not conducted a user study or established improvements in learning outcomes.

Disclosures

We use Google MediaPipe’s pretrained Hand Landmarker, Anthropic Claude through LangChain deepagents, and the open-source libraries listed in Built With.

The project uses Freeverb public-domain tunings. Our coordination board helper was adapted from azizu06/hackathon-agent-board. Fonts are used under their respective open-source licenses.

Claude Code, Codex, and Gemini generated substantial portions of the code under the team’s direction and review.

Simulated, scripted, animated, and replayed media are identified as such. The cover is an illustration, not a capture of a live rig.

Built With

Share this project:

Updates

Submission history