Resonance

Your voice knows how you feel before your conscious mind does.

Comment

What does it do?

Resonance is a private vocal reflection companion that transforms the raw acoustics of your spoken voice into living, generative procedural landscapes and actionable wellness insights.

It lets you helps relieve your overwhelming stress after a burnout day when you don't want to grab your diary and write but you just want to talk it out, you can just vent for how long you want and it will not only listen to you but also analyse how your voice is changing with the emotions you are facing. It helps you analyse your own self in a way that only you can't analyse yourself.

The user simply speaks their thoughts into the app, or even hums if the day was too tiring, and the app sorts the reflection into one of the terrains (visual representations of moods) and provides an (optional) mindfulness exercise for them. Sometimes a breathing exercise, sometimes a grounding one, depending on the given terrain.

There is an interactive prosody dashboard, which allows the user to reflect on the past week via a 7-day trend curve graph, as well as on the entire month through a color coded calendar. All the terrains have a procedural canvas wave display with a corresponding 'roughness' and color palette, so the mood type is immediately understandable to the user.

If they have a stable network connection, the user can also view real-time transcription via Web Speech API, in which case the transcription is also used in the selection of the terrain and logging further details.

The app is built as a privacy-first solution, because we believe your thoughts belong to you and only you get to decide who you want to share them with. The raw voice audio is discarded immediately after analysis, never saved to disk. All of the data is stored exclusively in local device storage, and the AI inference happens via an on-device JavaScript Multi-Factor Probabilistic Biome Classifier. Rather than a black-box model, Resonance uses five independent, hand-calibrated scoring functions one per biome, each built from acoustically meaningful thresholds. There is zero sign-up, zero log-ins and zero telemetry.

Inspiration

Traditional mood tracking and mental wellness tools can often be hard to use in moments of low energy or high stress. Some of the main issues we observed are:

  1. Cognitive Burden & Forced Labels: Selecting static emojis (😄, 😐, 😔) or typing long journal entries requires cognitive rationalization that often masks true underlying affect.
  2. Privacy Vulnerability: Users are hesitant to speak authentically into apps that stream raw voice recordings to remote cloud servers.
  3. Language & Literacy Barriers: Semantic sentiment analysis models struggle across multilingual speakers, dialects, or non-verbal vocalizations (sighs, hesitations, cadences).
  4. Lack of Somatic Feedback: Standard apps tell you what you logged, but fail to provide immediate, somatic self-regulation mechanisms.

We built this app to solve these issues and create a simple way to record and track mood and feelings. It reduces the required effort (e.g. writing, any form of journaling, choosing an exact mood for that moment) and allows the user to just speak their thoughts out loud.

Technologies we used

The app is front-end only, built primarily using React (v.19.2) and Vite (v.8.2.). The acoustic signal processing uses 3 main tools:

  • Web Audio API - a browser API for processing and synthesizing audio in real time,
  • Pitchy (YIN Algorithm) - a JavaScript pitch-detection library, estimating fundamental frequency from audio signal,
  • Meyda - a JavaScript audio feature extraction library

For the visuals and procedural graphics we used HTML5 Canvas 2D and Fractional Brownian Motion (fBm) noise algorithms. Some icons were taken from the Lucide React open-source library, and for styling we used CSS3.

The app is deployed on Vercel.

AI Usage

For coding assistance we used free tier AI models:

  • Anthropic Claude: Sonnet 5 and Haiku,
  • OpenAI ChatGPT: GPT-5.6 Luna.

For design prototyping we used free tier AI models:

  • Google Gemini: Nano Banana 2,
  • Figma Make.

Which track are you submitting to?

Wellness.

Team info

On this project we worked as a 4-person team:

  • Kusum S (Kusum0710): audio analysis/AI, recording pipeline, reflection/settings, dashboard, core integration.
  • Vinit M (Vinit-786): live terrain, breathing/grounding UX, recording UI, animation engine.
  • Mia Matić (miamatic04): home/history/timeline, monthly overview, export, Result Card/Day Detail, analyzer/settings refinements.
  • Rakshita Kumari (Rakshita006): settings, profile settings, and history management.

Challenges we ran into

1. Transcript duplication bug (Web Speech API)

The initial SpeechRecognition implementation rebuilt the entire transcript from scratch on every onresult event, including interim (non-final) guesses causing repeated and garbled text. Fixed by looping from event.resultIndex and only appending text when result.isFinal was true, separating "locked-in" transcript from "live preview" text.

2. Data shape mismatch between refactored files

A teammate refactored audioAnalyzer.js to return a single joined analysis string (insightMessage) instead of an array (analysisLines), while ResultCardModal.jsx still expected the old array format. This caused a silent failure at first (self-analysis note fell back to generic text with no crash), and later an outright crash (analysisLines.map is not a function) once the two files diverged further. Root cause: two people changing interdependent files without syncing the shared data contract. Fixed by aligning both sides back to a single agreed-upon shape.

3. Corrupted localStorage from a bad test session

While debugging, a session with undefined fields got saved to localStorage, which crashed the entire TimelineScreen on next load (one malformed object broke rendering for the whole component tree, not just itself). Lesson: no validation existed before persisting session data.

4. Web Speech API network failures

Live transcription intermittently failed with a network error since we were using Chrome's built-in Speech Recognition which depends on a live connection to Google's servers (not an on device model, despite feeling local). No API key was ever missing; this was a genuine connectivity/environment issue, not a config problem. Solved with a UI fallback so failures don't look like the app silently broke.

5. On-device Whisper (transformers.js) integration attempted

Tried adding fully local transcription as a network-independent backup. Hit a JSON.parse error caused by a fetch request resolving to the app's own index.html instead of the model's actual files a routing/bundler-level issue, not something fixable in the time available. Deliberately deprioritized in favor of keeping the tested, working Web Speech API path intact rather than risking the whole demo on an unfinished integration.

Accomplishments that we're proud of

We functioned well as a team, the work was well distributed based on our individual capabilities, everyone did their part and we could always find a compromise.

Using new technologies was a challenge, but we definitely learned a lot and will be able to apply a lot of this knowledge further in our projects. Most of us have never worked with audio processing and analysis so getting the analysis to work is a big win for us.

We built a functional on-device JavaScript neural network. Privacy has been a major concern for us from the start of this hackathon, so we put a great deal of effort into providing it for the user.

What's next for Resonance

The main thing we see as the next step would be changing the project from a web app to a mobile app, which would enable easier use and offline capability.

Some additional features that would greatly improve the user experience would be:

  • Additional accuracy - Implementing LLMS for more accuracy and better analysis with increasing cloud based storage.
  • Wearable Sensor Fusion: Correlate acoustic prosody scores with Apple Watch / Garmin real-time Photoplethysmography (PPG) Heart Rate Variability data.
  • WASM Acoustic Engine: Compile custom C++ DSP filters (Mel-Filterbank Cepstral Coefficients) into WebAssembly for sub-millisecond edge profiling.
  • Spatial Audio Sonification: Procedurally synthesize relaxing 3D spatial soundscapes reflecting the user's terrain in binaural audio.
  • Encrypted Decentralized Sync: Optional end-to-end encrypted backup using user-owned decentralized keys (Web3 / DID / passkeys).

Built With

Share this project:

Updates

Submission history