HEREBEAT

What if your phone's vibration motors can help you sense the beats of drums ? A site for deaf/hard-to-hear people to feel the music , because most affordable hearables cant produce lower frequencies.

Comment

Note :

You will need an Android device (tested on Android 13 and above) to use this.

Inspiration

Long ago had a friend , who was hard-to hear affected by congenital hearing loss . He was very fondly interested in playing music and drums , but he needed hearing aids and devices to hear media and sounds.

These devices has a frequency range from 100hz-2khz , and that too 100 hz for costly high tier ones . The affordable ones typically range from 200-2khz and basically drop all bass and beats .

We usually used to listen to songs over Spotify together and he wasn't able to enjoy the beats and map them out as i could .

At that time it was a promise of mine to him that i would make this to push this issue further along with a small start using a cell phones vibration motor further. This project is dedicated to him and similar people.

What it does

The project currently is a website where you can upload your music files , connect your google drive and store your songs . Then give the songs to the site , it separates different frequencies from the song and does source separation ( breaking the components of song eg. guitar , piano , drum etc .) .

then the frequencies are matched according to song for verification and classified using a classifier model Demucs (a neural model) along with statistical separation of frequencies using original equalizer algorithms.

After isolating frequencies their frequency and occurrence per second is counted and mapped out . This is compared to frequency response time of a phones vibration motor , (ERM/LRA) to match latency .

Latency matching is important because the motors of a phone can change speed under certain milliseconds only and over-pushing this latency will result in muddy output or the motor just ringing continuously .

After separating classifying and aligning the output to song standards the front-end UI displays the output to user. Separate options are added for saving songs and managing them at user end.

Front-end has two views , music and drumming view . Drumming view allows you to see graphs of songs , play beats and analyze the sound through feeling it properly along with the live sound of music through your phone haptics. While the music view serves as simple music player for all people to play music and enjoy it.

How we built it

This engine was built using two separate parts, front-end and back-end. The front-end was built mainly using React and Next.js and the back-end was built using Python and FastAPI.

The front-end communicated with the processing server through a same-origin API layer. This allows the application to keep the processing infrastructure separate from the public interface. The back-end was made as a modular pipeline so that the audio processing, analysis, haptic generation and storage can be developed independently.

For performance, the processing server keeps the heavy analysis components loaded rather than repeatedly initializing them for every request. Analysis results are represented as structured data so they can be reused later instead of forcing the entire processing pipeline to run again.

The application also uses a layered storage architecture. Google Drive is used as the user's persistent storage for songs and their generated analysis files, while the back-end database stores lightweight indexes and references. This keeps the actual media owned by the user while allowing the application to quickly find and restore previously processed songs.

The system was deployed with the Next.js application hosted on Vercel and the FastAPI processing back-end running on Google Cloud. Authentication, persistent sessions, encrypted credentials, Drive authorization and automatic back-end startup were integrated into the deployment so the system could operate as a continuously available service rather than only as a local prototype.

For back-end following stack was used :

  • Demucs v4 (htdemucs) — Deep learning model (Meta AI) for high-quality 4-stem audio source separation (Drums, Bass, Other, Vocals).
  • PyTorch (torch & torchaudio) — Deep learning tensor engine running CPU/GPU stem separation.
  • Essentia (essentia) — Music Information Retrieval (MIR) library implementing RhythmExtractor2013 (multifeature beat tracking, BPM, confidence, and meter detection).
  • librosa — Audio feature extraction, STFT spectrograms, onset detection, and spectral flux analysis.
  • NumPy & SciPy — Signal processing (Butterworth bandpass filtering, envelope detection, RMS calculation, transient tracking).
  • SoundFile & sounddevice — Multi-format audio I/O (WAV, FLAC, OGG, MP3, AAC, M4A) and audio device querying.
  • Matplotlib — Waveform, spectrogram, and diagnostic visualization generation.
  • FFmpeg — Multi-format media demuxing, decoding, and conversion.

Challenges we ran into

The task was very challenging as to build a full audio source separation engine is very hard , and i wanted it to be really usable so adding google authentication , making the google drive storage work , also the assets for the sites were hand-drawn and ai refined original drawings are uploaded on repository under /sorry folder :) .

Audio separation took me 2 -3 days of full dedication , sometimes bass separation not working , kicks in songs not being detected by the audio engine , or the beats tuning up too high . Almost at start with the Demcus model , bass and kicks weren't even getting detected and my ai agents were burning tokens like crazy . So i came up with audio of , if i separate the bass frequencies beforehand the input this can be tackled easily so , i used same sound separated into two streams , bass and main audio , the bass stream used equalizer to cut all higher frequencies so mapping it became nearly perfect matching 92 percent up-to bass beats .

Another challenge for me was creating hand drawn assets . I wanted the site to feel something a human can smile to . So i skipped normal AI heavy site designing and kept it simple with drawings. The color theming was another issue which i fixed using inverting colors on UI theme switch .

Now main engineering deploying hurdle was file dependent architecture . On newer phones with more memory the browser stays active while user chooses a file , but for a hard to hear or deaf person , its not necessary to have a well memory suitable phone , so i added file indexing, once file is chosen, it is indexed using duck db and uploaded later too so user doesn't looses file state .

Another main issue they wont relate it directly to hackathon was completing google auth flow in the deadline . From site verification to deployment Oauth implementation and other stuff , made it tougher and harder to move forward since API calls had to rely on some URL details . I used official domain etc because this meant more than a hobby project to me.

Accomplishments that we're proud of

I am mostly proud that this will help the deaf and hard - to - hear people get a reference of beats of songs while listening to song itself even transforming their experience of music and its art to something they may be able to feel .

What's next for HEREBEAT

For herebeat , next direction would be strengthening its back-end for scale , adding support not only for drummers , pianist , and people from other instruments as audio separation layer already implements it but adding their proper controls and exposure through UI would be next task .

If you have read this far , it was written by a human :) .

Built With

Share this project:

Updates

Submission history