Inspiration
I love listening to people talk about their experiences. One day that curiosity sent me down a rabbit hole: How do Deaf and hard-of-hearing (DHH) people experience music?
I’d seen ASL interpreters at concerts before, but that’s mostly about translating the meaning of lyrics. What about the feeling of the music? The bass drops, the build-up, the little details that make you close your eyes and go, “ohhh this part.”
So I started watching interviews and talks from DHH creators and community members. A pattern kept coming up: some people enjoyed feeling the bass through the floor or speakers, some focused on certain frequencies or vibrations, some preferred silence, some liked signed music, some didn’t. There was no single “DHH way” to listen to music—just like there’s no single “hearing way” either.
That stuck with me.
If everyone perceives music differently, and sensitivity is such a big factor, then accessibility can’t be one-size-fits-all. People should be able to customize how they experience it.
Haptica grew out of that thought: If we can’t assume one “right” way to listen, could we at least build another option— a small haptic translator that lets someone feel parts of the music, in a way they can tune for themselves?
It’s not a final answer, but it’s my first step toward “music for all” meaning more than just speakers and earbuds.
What it does
Haptica is a tiny ML-powered haptic translator for music.
It takes audio input, runs it through a lightweight neural network, and outputs vibration patterns on coin motors so people can feel changes in the music: like shifts in energy, sections, or rhythm, instead of only hearing them through speakers or earphones.
How we built it
Collected and prepared music/audio samples, then extracted compact audio features. Designed and trained a small depthwise-separable convolutional model in Google Colab. Converted the trained model to a TensorFlow Lite (.tflite) file to fit on a microcontroller. Deployed the model onto an Arduino Nano BLE Rev 2, chosen for low power and tight memory. Drove coin vibration motors from the Nano and mapped model outputs to simple haptic patterns (pulses, on/off, intensity changes) that reflect what’s happening in the music.
Challenges we ran into
Getting a neural network small enough (in RAM and flash) to run on the Nano without crashing. Making the TFLite model export and Arduino integration play nicely together. Debugging wiring + motor control while the board was also doing inference. Figuring out which musical features actually feel meaningful when turned into vibration, without overwhelming or under-stimulating the user.
Accomplishments that we're proud of
Turning a curiosity rabbit hole about DHH music experiences into a working physical prototype.
What we learned
Deaf and hard-of-hearing people are not a monolithic group; there is no single “DHH way” to experience music. Accessibility is as much about options and personalization as it is about technology. Tiny ML on constrained devices forces you to be intentional about model size, data representation, and every extra line of code. Hardware + ML + human experience is messy but rewarding, you cannot just ship a model; you have to think about how it feels.
What's next for Haptica
Co-design and test with DHH users to understand what patterns actually help and what just feels annoying or noisy. Add more expressive haptic vocabularies (different motors, locations, and rhythms for different musical elements). Let users tune sensitivity and intensity, since everyone’s comfort and perception levels are different. Explore integrations with phones or media players so Haptica could eventually plug into real-world listening setups, not just a demo.
Log in or sign up for Devpost to join the conversation.