Aphasia Agent

• 

Inspiration

I work as a researcher for inclusive data visualization and worked with people with aphasia. When I do the data analysis from video, I had to annotate the speech because the usual transcriber could not get the correct utterances from people with severe aphasia. People with non-verbal aphasia (very severe case of aphasia or with non-fluent or Broca’s aphasia) often communicate through co-speech compensatory gestures (iconic, deictic, pantomimic) paired with fragmented vocalizations, phonemic paraphasias, and word-finding delays. Annotating these video sessions manually very tedious, taking up to 10–15 hours per two hour of video.

What it does

AphaLink automates this entire heavy workflow:

Multimodal Ingestion & Audio Resampling: Ingests raw video/webcam recordings, extracts high-fidelity visual keyframes, and decodes the audio track into 16kHz mono PCM WAV format.

Synchronized 3-Tier Clinical Annotation: Leverages Gemini’s multimodal vision and audio understanding to generate a synchronized ELAN-style timeline:

• Tier 1 (Speech / Utterances): Transcribes verbal output, annotates pauses (>1.5s), and flags phonemic approximations.
• Tier 2 (Gesture Kinematics): Categorizes hand and arm movements using the standard McNeill Gesture Taxonomy (Iconic, Deictic, Beat, Metaphoric) with spatial phase tracking.
• Tier 3 (Synthesized Communicative Intent): Cross-references speech and gesture cues to decode the patient’s intended message.

Automated Clinical SOAP Note & Analytics: Automatically computes clinical metrics (Words Per Minute, Gesture Compensation Ratio, Utterance Success Rate) and compiles standard Subjective, Objective, Assessment, and Plan (SOAP) documentation.

Longitudinal Adaptive Lexicon: Identifies recurring non-verbal gesture patterns and logs them into a patient-specific dictionary for longitudinal therapy tracking.

Interactive AAC Communication Deck: Provides an interactive word-finding aid with auditory playback and quick-response cards to assist patients in real time.

How we built it

AI & Multimodal Engine: Built with Google’s Gemini API using the official @google/genai TypeScript SDK. I designed structured JSON schema prompts that pass synchronized keyframe images and resampled raw audio bytes simultaneously to perform joint audio-visual temporal alignment.

Client-Side Signal Processing: Implemented browser-side video frame extraction using HTML5 Canvas and developed an in-browser Web Audio API pipeline to extract and resample audio directly into 16kHz mono WAV chunks.

Backend Architecture: Express.js server in TypeScript with resilient multi-model routing (gemini-3.1-flash-lite, gemini-2.5-flash, and gemini-3.7-flash) to handle variable request sizes and quota limits without interrupting clinical workflows.

Frontend & Visualization: Built in React 18, TypeScript, and Tailwind CSS. I created an interactive interactive 3-tier timeline viewer, waveform visualizer, exportable SOAP clinical report generator, and an adaptive dictionary manager.

Challenges I ran into

The gesture worked quite well in the first few iteration. However, the speech recognition was more difficult to handle. Speech recognition standard models often hallucinate or fail entirely on severe dysfluent speech or non-lexical sounds. I addressed this by combining direct acoustic listening with visual gesture context in the Gemini prompt so the model can disambiguate intent using cross-modal compensation. In the 20 second test, it detected 3 out 4 corrected speech. It might need a better speech recognition system but it might be challenging for people with aphasia.

Building agentic workflow. I didn't know what agentic means and had to learn from the resources. I wasn't sure how to use ADK as well, and having the possibility of exceeding the provided credit is quite daunting.

Accomplishments that I'm proud of

Building a system that successfully bridges the gap between what a patient says and what their hands gesture. The gesture and intent soft of work well. Intent is a bit more abstract but gesture handled well.

What I learned

The Value of Multimodality in Healthcare: Aphasia communication cannot be solved by audio or video alone. Gemini’s unified vision and audio understanding is critical for clinical scenarios where verbal cues are incomplete or compromised.

Efficient Client-Side Audio/Video Preprocessing: Resampling audio to 16kHz PCM WAV in the browser drastically reduces backend network payload while preserving the exact acoustic fidelity needed for speech-language analysis.

Structured Schema Enforcements: Using responseSchema with the @google/genai SDK is game-changing for converting chaotic multimodal inputs into consistent, type-safe clinical timelines.

What's next for Aphasia Agent

Multilingual & Dialectal Gesture Recognition: Expanding the adaptive lexicon to support cross-cultural gesture repertoires and regional sign variations.

Clean up cluttered UI

Test with more people with aphasia

Built With

• gemini
• typescript
• vite