Rimay Adaptive Voice Coach
Inspiration
After a stroke, some people may need structured opportunities to practice speech production. However, many digital tools are difficult to follow, depend on external services, or present scores that can easily be mistaken for clinical evaluations.
We wanted to explore a different approach: a clear and accessible browser-based experience where a person can practice words, short phrases, and guided reading while remaining in control of every step. The project was designed around three principles: privacy, accessibility, and transparency.
Rimay Adaptive Voice Coach is a technical demonstration, not a medical device. It does not diagnose dysarthria, classify severity, prescribe treatment, or replace a speech-language professional. Its purpose is to demonstrate how local browser technologies and deterministic rules can support a structured practice experience without sending recordings to an application backend.
What it does
Rimay guides the user through a session of five valid practice attempts. The session begins with word repetition, continues with phrase repetition and guided reading, and then selects later exercises using a deterministic adaptation policy.
For every exercise, the user can:
- Read or listen to the instruction.
- Record a temporary audio attempt.
- Use optional browser speech recognition or manually enter what they attempted to say.
- Review locally calculated acoustic and textual metrics.
- Receive short, non-clinical feedback.
- Repeat the exercise or continue to the next one.
The application offers three input modes:
Browser recognition
Rimay can use SpeechRecognition or webkitSpeechRecognition when supported. Recognition is optional because browser implementations vary and may rely on services managed by the browser vendor.
Manual input
The user can type the words they attempted to pronounce. Rimay clearly identifies this as user-declared text and does not claim that it was verified against the recording.
Deterministic demo
The demo mode uses local fixtures and requires no microphone, speech recognition, or network connection. It provides a reproducible way to demonstrate the complete five-attempt session.
A blocking capture, such as silence, insufficient voice activity, clipping, or an extremely short recording, does not count toward session progress. The user can repeat the attempt or continue without registering it as valid.
How we built it
Rimay is a static single-page application built with:
- React
- Vite
- TypeScript
- Tailwind CSS
- Web Audio API
- MediaRecorder
- Web Speech API
- Vitest and Testing Library
The runtime does not use OpenAI, Supabase, a backend, serverless functions, or commercial APIs.
The main processing flow is:
temporary recording or demo fixture
→ local acoustic metrics
→ optional textual result
→ local textual metrics
→ deterministic coaching rules
→ feedback and next action
Local acoustic analysis
The recording is decoded using the Web Audio API. audio-metrics-v1 calculates technical observations such as:
- total duration;
- estimated voice activity;
- RMS level;
- peak amplitude;
- silence ratio;
- internal pauses;
- possible clipping.
These values are experimental interaction metrics. They are not clinical scores.
Local text analysis
When final text is available, text-metrics-v1 normalizes Spanish text, aligns words, and calculates matches, substitutions, omissions, additions, word error rate, similarity, and words per minute when the available evidence is sufficient.
The comparison remains local and preserves the source of the text: browser recognition, manual input, or demo.
Deterministic coaching
Instead of generating feedback with an AI service during runtime, Rimay uses coach-rules-v1 and a catalog of curated templates.
The same input always produces the same decision. Every result includes a rule identifier, template identifier, explanation, and supporting evidence. The system can only select exercises from the local approved catalog.
This made the application easier to test, audit, and demonstrate without depending on network availability or model variability.
State machines
We separated the lifecycle of an individual attempt from the lifecycle of the complete session.
The attempt controller manages recording, recognition, text input, analysis, feedback, errors, and cleanup. The session controller manages accepted attempts, exercise coverage, adaptation, progress, completion, and starting a new session.
An attempt is added to the session only after the user explicitly presses Continue or Finish session. Analysis alone never changes the session history.
Challenges we faced
One of the largest challenges was the inconsistent behavior of browser speech technologies. Speech recognition did not always produce a usable final result, so we treated it as progressive enhancement and kept manual input available as a reliable fallback.
Speech synthesis also required careful lifecycle management. Some browsers initially returned an empty voice list and loaded Spanish voices later. We added synchronization around voiceschanged so the instruction button becomes available without requiring the user to leave and return to the screen.
Audio recording introduced another browser-specific challenge. Chromium could produce live WebM recordings without complete duration metadata. The audio element could sometimes load them, while decodeAudioData rejected the same file. We improved recorder finalization, isolated recording sessions, waited for final dataavailable events, and prioritized MP4/AAC where supported while retaining WebM as a fallback.
We also had to prevent late asynchronous events from affecting a new attempt. Generation tokens, monotonic attempt identifiers, explicit cleanup, and immutable snapshots were used to avoid duplicate registrations and stale updates.
A product challenge was maintaining a clear boundary between technical feedback and clinical interpretation. The interface describes the current capture but never labels the user, estimates severity, predicts recovery, or recommends treatment.
What we learned
We learned that browser APIs can provide a capable zero-cost foundation, but they require defensive engineering. Support detection alone is not enough; timing, permissions, delayed events, codecs, cleanup, and vendor-specific behavior must all be considered.
We also learned that deterministic systems are valuable when explainability and reproducibility matter. Versioned rules and fixtures allowed us to test exact boundaries, repeat decisions, inspect evidence, and demonstrate the application without relying on a remote model.
Privacy influenced the architecture from the beginning. The audio exists only in memory during the active attempt. Object URLs, media tracks, recognition state, and temporary results are released when the user repeats, continues, finishes, or leaves the flow. The submitted version has no persistence, telemetry, or application-controlled upload of audio or text.
Accessibility was not treated as a final visual adjustment. Instructions and feedback are available as visible text, audio never starts automatically, controls are keyboard accessible, focus is managed during major transitions, and guided-reading pauses are represented with text rather than color alone.
How we used Codex
Codex and GPT-5.6 were used as development and review tools, not as runtime services.
They helped us:
- Convert product requirements into typed contracts.
- Plan the project in small, verifiable increments.
- Review architecture and browser-specific risks.
- Design deterministic fixtures and adversarial test cases.
- Identify race conditions and stale asynchronous events.
- Audit privacy and accessibility constraints.
- Reconcile documentation with the implemented behavior.
- Validate each increment before creating a local commit.
The final application does not call an OpenAI model during recording, analysis, coaching, or adaptation.
Accomplishments
We are particularly satisfied that Rimay delivers a complete five-attempt adaptive session while remaining:
- fully browser-based;
- usable without a backend;
- reproducible through demo mode;
- transparent about the origin of text;
- explicit about non-clinical limitations;
- tested through 361 automated tests;
- deployable as a static Vite application.
What’s next
The current version intentionally focuses on a stable participant experience. Future work could include:
- a larger exercise catalog;
- sanitized local persistence;
- local participant and professional navigation roles;
- a deterministic technical session summary;
- additional accessibility testing with screen readers;
- evaluation of the exercise content with speech-language professionals;
- broader validation across browsers, devices, microphones, and Spanish voice configurations.
Any future clinical use would require professional review, formal validation, stronger data governance, and a substantially different product and regulatory process.
Built With
- accessibility
- assistive-technology
- css3
- digital-health
- gpt-5.6
- healthtech
- html5
- javascript
- mediarecorder
- openai-codex
- privacy
- progressive-web-app
- react
- speech-therapy
- speechrecognition
- speechsynthesis
- stroke-rehabilitation
- tailwind-css
- testing-library
- typescript
- vercel
- vite
- vitest
- web-audio-api
- web-speech-api
Log in or sign up for Devpost to join the conversation.