Getting data to train accuracy evaluation model on
Wiring Mayhem

Tactus

Inspiration 🎸

Our teammate Aditya loves DJing and playing guitar, and wants to help bring the language of music to the world 🎶. That led us to a bigger question: what if learning an instrument did not depend on hearing at all? Instead of treating accessibility as an add-on, we wanted to build a system where touch, sight, and AI work together as the primary teaching interface.

We were also excited by the technical challenge. Guitar playing is hard to interpret from a camera and a mic alone: fingers occlude the fretboard, buzz does not directly tell you its cause, and useful feedback has to be immediate and specific. That made Tactus the kind of hackathon project we love: meaningful, ambitious, hardware-heavy, and legitimately difficult.

What it does 🎯

Tactus is a Deaf-accessible guitar coaching system. It watches your fretting hand with a webcam, listens to your guitar with a mic, fuses those signals to infer what note you played, where you played it, which finger you used, and how cleanly you fretted it, then turns that into one precise correction you can feel on your body and see on screen.

It has two core modes:

LEARN 🎓: feel the target song on a 12-channel haptic wearable, play it on a real guitar, and get corrective feedback when you make a mistake.
PLAY 🎵: get real-time tactile guidance for pitch, timing, and correctness while playing freely.

The feedback loop is the key idea. When a player makes a mistake, Tactus can replay the correct haptic target, compare the player's attempt against the reference, highlight the relevant finger, and present a simple physical correction instead of vague advice. The goal is to recreate the "hear it, fix it" loop as a "feel it, see it, fix it" loop ✨.

Redis is an important part of that experience. We use Redis as the local-first memory layer for practice: storing analyzed song references, rendered haptic patterns, target fingering, calibration, and user progress so practice sessions can run offline 📦. That matters because a coaching device should still work in a basement, rehearsal room, or hackathon venue with unreliable internet. Redis also gives us a path toward retrieving similar past mistakes so the system can say, in effect, "this is like the buzz you made last time," which opens the door to adaptive coaching rather than one-off corrections.

How we built it 🛠️

We split the system into two halves running on a single laptop 💻.

The browser side handles camera capture, the visual interface, and hand/fretboard understanding using MediaPipe Hands and geometry-based fretboard mapping 👁️.
The Python side handles mic input, pitch tracking, feature extraction, fusion logic, and 12-channel haptic rendering 🎛️.

Those two halves communicate over a localhost WebSocket 🔌. That let us develop the visual system and the audio/haptics engine in parallel while keeping a clean contract between them.

On the hardware side, we built a 12-channel wearable haptics rig using USB multichannel audio adapters, class-D amp boards, and de-housed speaker drivers mounted on the body 🦺. Six channels map to the strings across the back, and six channels map to fret zones across the torso. Instead of generating an abstract score, we render music and corrections directly into vibration patterns.

Tactus system architecture diagram

On the perception side, we combined:

audio analysis for pitch, onset/offset timing, and note quality cues 🎧
vision features from the fretting hand and fretboard geometry 👋
fusion logic that reconciles what the camera sees with what the mic hears 🧠
an LLM-based coaching layer for phrase-level natural-language feedback 💬

Tactus ML inference pipeline

We also designed Redis into the architecture as more than a cache. Redis acts as Tactus's memory. It stores the offline song library, user calibration, progress, and historical error context. In our design, Redis also supports nearest-neighbor retrieval on feature vectors so we can surface similar past mistakes and make coaching more personalized over time. That gave us a strong sponsor fit, but more importantly, it fit the product honestly: a wearable practice coach should remember your songs, your body, and your progress.

Challenges we ran into 🚧

The biggest challenge was that almost every interesting part of the project sits in an awkward real-world boundary.

Hardware was one 🔩. Building a wearable 12-channel haptics system quickly meant dealing with amplifier wiring, body coupling, actuator placement, power constraints, and the fact that something can be audible, visible, and physically underwhelming at the same time. Making vibrations feel distinct on the body is a much harder problem than simply making something vibrate.

Perception was another 👀. Guitar technique is hard to read from raw signals. A bad note, weak pressure, poor placement, and bad timing can overlap, and the camera's view of the fretboard is often partially occluded by the player's own hand. We had to be disciplined about putting AI only where the problem is genuinely unsolved and keeping the rest of the system deterministic and measurable.

Latency and architecture were also real constraints ⏱️. Immediate haptic feedback needs to feel responsive, while richer fused corrections arrive a bit later. That forced us to separate fast feedback from deeper coaching instead of pretending one model could do everything instantly.

Finally, Redis introduced an honesty challenge in the best way: if we were going to talk about memory, it needed to be a real product function. That pushed us to think carefully about what must persist locally for a practice session to work offline and what kinds of retrieval would actually help a player improve, rather than adding a database just to satisfy a sponsor category.

Accomplishments that we're proud of 🏆

We are proud that Tactus is not just an accessibility pitch deck. It is a coherent system with a real hardware rig, a grounded software architecture, and a clear theory for how tactile coaching can work 💡.

We are proud of the product framing: instead of claiming to "make Deaf people hear," we built around a more respectful and rigorous idea, creating a new sensory pathway for learning music through touch and vision.

We are proud of the technical split between solved and unsolved problems. The system uses ML and multimodal fusion where it earns its keep, then hands off to deterministic haptic rendering so the output remains inspectable and honest.

We are also proud of the Redis story. We did not treat Redis as a generic cache. We treated it as the memory of the coach: the offline library, the calibration store, the progress tracker, and eventually the retrieval layer for similar mistakes and adaptive difficulty. That made the architecture stronger and the demo more believable.

What we learned 📚

We learned that accessibility projects get much better when they are designed around a user's actual learning loop instead of around a flashy output. The most important insight was not "how do we visualize music," but "how do we let someone notice and correct a mistake without relying on hearing?"

We learned that haptics is unforgiving. Frequency choice, actuator coupling, body placement, timing, and intensity all matter, and small physical design changes have huge perceptual effects.

We learned that multimodal AI becomes much more credible when each modality has a clear job. Vision is good at telling us where the hand is and which finger is involved. Audio is good at telling us what actually sounded and how cleanly it sounded. Fusing them gives us a better correction than either alone.

We also learned that Redis is especially compelling in embodied systems. Memory is not just a backend feature; it changes the user experience. When a system remembers your calibration, your songs, and your recurring mistakes, it starts to feel like a real coach instead of a stateless demo 🧠.

What's next for Tactus 🚀

We want to expand the offline library, support more songs cleanly, improve the wearable form factor, and test with more real players to further develop our teaching and feedback methodology 🎸.

Longer term, we see Tactus becoming a broader platform for accessible instrument learning: still rigorous, still tactile-first, and still grounded in the idea that the language of music should be available to more people.