DoingLanguage

Speech Synthesis

DoingLanguage : Agentic Speech & Language Rehabilitation

Empowering stroke survivors and clinicians with an adaptive, multi-modal therapeutic co-pilot. Submitted to the All Things Agentic Hackathon.

💡 Inspiration

You were Chief Technologist with extensive expertise in cloud systems, service-oriented architecture (SOA), digital design, and artificial intelligence. Throughout your career, you have delivered end-to-end architectures, strategic platforms, and AI prototypes across major commercial and enterprise organizations, including Accenture Australia, South East Water, National Australia Bank, and CogniVirtus. Backed by a foundational double-degree in Computer Science and Instrumental Science from Swinburne University, your work bridges technical innovation, industry standards and patented technology solutions.

In May 2024, my life changed overnight when I sustained a left middle cerebral artery (MCA) embolic stroke. Coming on the heels of surgery for localized prostate cancer and complicated by hypertension and post-operative fatigue, the physical aftermath was daunting: reduced fine dexterity in my right hand, mild lower-limb weakness with functional foot-drop, and, most acutely, residual dysphasia and apraxia of speech. My thinking has also slowed (1/10 of what it was can't remenber some the finer datails. EG i cant program any more but can think architecturally , thus this ppproject.

While I fought hard to remain independent in personal care and mobility, communicating thoughts that were crystal clear in my mind felt like shouting through a soundproof barrier.

Working through Australia’s National Disability Insurance Scheme (NDIS) to optimize therapy access revealed a universal clinical bottleneck: the Therapy Cliff. A dedicated Speech-Language Pathologist (SLP) might provide 1–2 hours of clinical therapy per week, but neuroplastic recovery and the Principles of Motor Learning (PML) demand intensive, high-volume repetition:

Repetition Target: R≥50 to 100 active productions per target/session

Between weekly appointments, patients are left with static paper worksheets, unengaging flashcards, or apps with harsh countdown timers that induce cognitive fatigue.

DoingLanguage was born from a direct lived need: to build an agentic, multi-modal therapeutic co-pilot that bridges the home-clinic divide, enabling patients and SLPs to collaborate seamlessly in driving neurological recovery.

🎯 What It Does

DoingLanguage is an accessible, offline-capable Progressive Web App (PWA) powered by an autonomous therapeutic agent loop that guides stroke and brain-injury survivors through structured, progressive speech rehabilitation.

The 10-Tier Evidence-Based Curriculum The platform organizes language recovery into an unbroken hierarchy of 10 progressive tiers:

Tier 1–3 (Foundations & Words): Phonemes, letters, high-frequency monosyllabic words, and initial/final consonant blends. Tier 4–7 (Structure & Grammar): Compound words, multi-syllabic targets, carrier phrases, and structured syntax. Tier 8–10 (Autonomy & Discourse): Functional conversational exchanges, descriptive narratives, and abstract reasoning.

Patient & SLP Collaborative Synergy For the Patient: A calm, zero-pressure interface with large touch targets (designed for hemiparetic or reduced right-hand dexterity). No countdown timers, no failing buzzers, and offline resilience for practice anywhere. For the SLP: An intelligent agent dashboard that monitors acoustic precision, phonetic distortion rates, session adherence, and response latency. Clinicians can remotely tailor target word lists and export objective data for NDIS plan renewals.
Real-Time Multi-Modal Agentic Cueing The agent dynamically modulates its assistance according to the clinical cueing hierarchy: Assistance Level: Full Audio/Visual Model → fading Phonetic/Mouth Cue → fading Independent Production Assistance Level: Full Audio/Visual Model fading Phonetic/Mouth Cue fading Independent Production

🛠️ How We Built It

DoingLanguage was engineered as a high-performance, zero-bloat web application prioritizing instant loading, universal accessibility, and robust offline capability.

Client Core & Architecture: Built with vanilla ES modules, custom web components, and modular CSS with container queries—achieving sub-second load times without heavy framework overhead. Speech Processing & Agent Loop: Utilizes the Web Speech API (SpeechSynthesis and SpeechRecognition) coupled with an agentic scoring heuristic that calculates phonetic similarity using string-distance algorithms and acoustic duration matching.

Offline-First Data Layer: Powered by Service Workers and IndexedDB for local storage of exercise banks, user audio waveforms, session metrics, and customizable vocabulary.

Fatigue-Aware Adaptive Pacing: The system monitors response latency and error clustering to detect post-stroke cognitive fatigue, autonomously suggesting breaks or stepping down exercise complexity.

🚧 Challenges We Faced

Atypical Speech Recognition: Commercial Speech-to-Text (STT) models are trained on fluent speech and frequently misinterpret dysphasic or apraxic speech (prolongations, sound substitutions, dysfluent pauses). We designed custom phonetic tolerance thresholds and multi-modal fallback verification to prevent patient frustration.

Accessible Physical UI for Hemiparesis: Due to right-hand motor deficits common in left-MCA stroke survivors, standard UI components (small dropdowns, multi-finger gestures) cause fatigue. We designed large single-touch activation zones, high-contrast layouts, and full keyboard/switch navigation. Clinical Rigor vs. UX Simplicity: Balancing comprehensive clinical data collection for SLPs with a clean, stress-free interface for stroke survivors required extensive iterative design.

🏆 Accomplishments We're Proud Of

Zero-Pressure Motor Practice: Created an empathetic digital environment where patients can perform the necessary 50+ repetitions per session without anxiety or penalty.

True Offline Autonomy: Fully functional as a standalone PWA, enabling uninterrupted rehabilitation in hospital rooms, rural clinics, or at home.

Quantifiable Clinical Telemetry: Built structured progress reporting that turns home practice into actionable clinical data for NDIS reviews and SLP goal setting.

📚 What We Learned

The Power of Micro-Tiers: In stroke recovery, a leap from single words to 3-word phrases can be a wall. Breaking the curriculum into micro-steps (carrier phrases, rhythmic pacing, visual syllable splits) sustains momentum.

Agents as Empathetic Partners: Autonomous agent loops in healthcare must not act as rigid evaluators; they must serve as patient, encouraging partners that adapt to the human being behind the screen.

🚀 What's Next for DoingLanguage

With funding from the All Things Agentic Hackathon, we will scale DoingLanguage from an innovative prototype into a clinically validated rehabilitation ecosystem:

On-Device Fine-Tuned Speech Models: Train lightweight, edge-run audio models specifically calibrated for dysarthric and apraxic speech patterns.

Two-Way SLP Tele-Rehab Portal: Launch real-time prescription modules where SLPs can push custom video prompts and patient-specific vocabulary (family names, daily essentials) directly to the patient app.

Formal Clinical Pilot: Partner with stroke rehabilitation centers and NDIS providers in Australia to run a 6-month longitudinal study measuring speech recovery acceleration against standard care.

Multilingual & Dialect Support: Expand phoneme libraries to support diverse accents, starting with Australian and Commonwealth English variations.

😰 Out of time

I only found out about this recently and am out of time - there are 3 "Try it out" links

The description of this project
The almost running cloud version
The running GitHub version

The 3 running sometimes works.

This is a miracle - my thinking has severely deteriorated. I'm up for talking about free credits and other things to carry on this work, possibly with some assistance.