Idiolect: Any Voice, Understood

• 

• 

• 

Inspiration

For many people with ALS and other conditions that cause dysarthria, speech recognition fails precisely when voice access matters most. Typing or using a touchscreen may also become difficult, making voice an essential interface rather than a convenience.

General speech models particularly struggle with personal words: a caregiver’s name, a street, or a medication such as “baclofen.” Existing personalized systems can require long enrollment sessions and scripted recordings. That is a serious burden for someone whose speech is tiring or changing.

We asked a different question: what if personalization happened through ordinary use? A person already corrects a failed transcript. Idiolect turns that correction into memory. Correct a word once, and the system can use it the next time.

What it does

Idiolect is an on-device speech-correction system for people with dysarthria, including people with ALS.

The interaction is simple:

1. Speak naturally.
2. Review several possible transcriptions.
3. Accept one or correct a word.
4. Idiolect remembers the correction as personal vocabulary.
5. Future transcriptions use that vocabulary and recent conversational context.

There is no scripted enrollment and no model-training delay. Names, medications, places, and other personally important terms become available immediately.

All speech, corrections, and personal vocabulary remain on the user’s computer.

How we built it

Idiolect combines specialized models rather than asking one model to do everything.

Parakeet TDT 0.6B, running through MLX on Apple Silicon, produces multiple candidate transcriptions. When those candidates disagree, Gemma 4 E4B’s native-audio capability provides a selective second opinion.

Gemma 4 then evaluates the candidates using the speaker’s personal vocabulary and their three most recent utterances. It may select an existing candidate or repair a supported near-miss.

Personalization uses contextual biasing rather than fine-tuning. The acoustic model remains fixed; corrections immediately become context for future requests.

We built the application in Jac/Jaseci. The Jac graph is the system’s memory:

• Profile, Session, and Utterance nodes represent use over time.
• Candidate nodes preserve alternative transcriptions.
• Correction and VocabEntry nodes capture what the system learns.
• Jac walkers transcribe audio, rerank candidates, accept corrections, manage vocabulary, and export training pairs.
• The client interface is also written in Jac.

The complete system runs locally through Ollama, MLX, and Apple Metal. It requires no hosted inference API.

Challenges we ran into

Our first reranker became less accurate as it learned more vocabulary. Given too many unrelated personal words, the model could construct a fluent sentence that was unsupported by the audio.

We addressed this with explicit safety constraints. Vocabulary must resemble evidence in the candidate transcripts, words shared by every candidate must normally survive, and newly introduced words must come from the candidate set or relevant personal vocabulary.

We also learned that not every spelling correction belongs in a language model. Near-misses such as “gaviscan” for “Gaviscon” are handled by a deterministic spelling pass, restricted to uncommon personal terms such as names, places, and medications.

Another subtle issue involved Gemma’s reasoning mode. Without think: false, the model could consume its output budget internally and return no usable answer. That looked like poor model performance but was actually a client configuration problem.

Finally, both Parakeet and Whisper normally discard their alternative beam-search hypotheses. We preserved those alternatives without reimplementing beam search, giving Gemma more acoustic evidence to reason over.

Accomplishments that we're proud of

We delivered a working local loop: speak, review candidates, correct a word, watch the personal vocabulary grow, and use that word successfully in a later utterance.

On a constructed personal-vocabulary benchmark using four synthetic, non-dysarthric proxy voices:

• Overall word error rate fell from 0.258 to 0.081 after 17 corrections: 69% fewer errors.
• Error on personal terms fell from 0.593 to 0.074: 87% fewer errors.

These figures demonstrate the personal-vocabulary mechanism; they are not clinical results.

We also evaluated real dysarthric speech from TORGO. We measured no meaningful improvement because its phonetically balanced prompts contain almost no recurring personal vocabulary. Importantly, the constrained system did not meaningfully degrade speech it could not improve.

We are proud of reporting both results instead of presenting the proxy benchmark as evidence of effectiveness for people with ALS.

What we learned

Personalization helps only when the transcript retains evidence of the intended word. “Baclofen” becoming “battle fan” may be recoverable; an unrelated substitution may not be.

We also learned that public dysarthric-speech datasets are poorly suited to evaluating personal vocabulary. They contain standardized prompts rather than the names, medications, relationships, and places that matter in a person’s daily life.

Most importantly, assistive speech technology needs a different definition of accuracy. Correctly recognizing a medication or caregiver’s name may matter far more than recognizing another common word.

What's next for Idiolect

The next step is participatory testing with people who have ALS and other speech disabilities, alongside speech-language pathologists and caregivers. We want to learn which correction interactions remain usable as speech and motor access change.

We would then:

• Evaluate people speaking their own personally meaningful vocabulary.
• Support switch access, eye-gaze navigation, and larger touch targets.
• Track voice changes over time without demanding repeated enrollment.
• Add optional pronunciation samples for especially important terms.
• Explore multilingual and code-switching support.
• Measure latency, fatigue, correction effort, and communication success—not only word error rate.
• Conduct accessibility, privacy, and clinical-safety reviews before presenting Idiolect as an assistive product.

Idiolect is currently a research prototype, not a medical device. Our goal is to build the next version with the people it is intended to support.

Built With

• gemma
• jac
• jaseci
• mlx
• ollama
• python
• speech