TheraVoice

An open-source, non-diagnostic assistive speech/communication monitoring companion

Comment

Inspiration

Communication can change from day to day, while clinical follow-up often happens only periodically. Changes such as slower speech, increased pauses, hesitation, repetition, or reduced pitch variation may go unnoticed between appointments.

We built TheraVoice to explore a different approach: instead of comparing someone with a population-level "normal," TheraVoice builds a personal baseline from the individual's own communication patterns and tracks meaningful changes over time.

Parkinson's disease is our primary motivating example, but the architecture is intentionally disease-agnostic. The goal is not to diagnose a condition, but to provide descriptive, longitudinal signals that can help a person understand changes in their own communication and optionally share those observations with a clinician.

Privacy and safety were also central to the idea. TheraVoice does not fabricate unavailable measurements, does not automatically change medications, and does not turn observations into diagnoses.

What it does

TheraVoice is an open-source, non-diagnostic assistive speech and communication monitoring system.

It provides an end-to-end pipeline:

Audio / Transcript → Biomarkers → Personal Baseline → Change Detection → Context → Multi-Agent Analysis → Assistive Actions → Longitudinal Reports

Key capabilities include:

  • Speech and language biomarker extraction

    • Speech and articulation rate
    • Pause patterns
    • Pitch and pitch variation
    • Voice intensity
    • Voice-quality descriptors
    • Hesitation and repetition
    • Sentence and lexical metrics

  • Personal baseline modeling

    • Builds a rolling baseline for each individual
    • Requires sufficient observations before detecting deviations
    • Never substitutes a population baseline when personal data is unavailable

  • Change and anomaly detection

    • Identifies deviations from the person's own historical patterns
    • Reports insufficient data explicitly rather than manufacturing conclusions

  • Multi-agent architecture

    • Speech analysis
    • Text analysis
    • Context understanding
    • Medication context
    • Therapy support
    • Longitudinal summaries

  • Optional LLM integration

    • Supports OpenAI and Google Gemini
    • Used only for natural-language summary generation
    • Structured biomarker extraction and event detection remain deterministic
    • Requires explicit patient consent for LLM processing
    • Falls back to deterministic summaries when the provider is unavailable

  • Bee integration

    • Supports real Bee data through Bee's documented local interfaces
    • Includes sync, proxy, and mock modes
    • Can analyze conversations without requiring a Bee account through the included demo

  • Privacy-aware processing

    • Consent-controlled storage and audio analysis
    • Raw audio is not persisted by default
    • Sensitive operations are audited without storing raw transcripts or secrets

  • Web dashboard and API

    • FastAPI backend
    • Patient timelines and biomarker views
    • Daily and clinical-oriented summaries
    • Bee synchronization
    • Transcript analysis
    • Medication and therapy tracking

Most importantly, TheraVoice is observational, not diagnostic. It describes changes and surfaces evidence; it does not determine that a person has a disease or automatically prescribe treatment.

How we built it

We designed TheraVoice as a modular pipeline so that individual components can be replaced without rebuilding the entire system.

The backend is built with Python 3.12+, FastAPI, Pydantic, and SQLAlchemy, with a React/Vite dashboard and optional AWS adapters.

The core analysis flow is:

Input
  ↓
Data Ingestion
  ↓
Preprocessing & Normalization
  ↓
Speech / Text Biomarkers
  ↓
Personal Baseline
  ↓
Change Detection
  ↓
Context Engine
  ↓
Multi-Agent Orchestrator
  ↓
Assistive Actions
  ↓
Timeline / Daily / Clinical Reports

We deliberately separated deterministic analysis from generative AI.

The biomarker system produces structured evidence and explicitly tracks whether a measurement is available. The baseline and change-detection layers operate on that evidence. The agent system then interprets the available information without inventing observations.

An optional LLM layer sits at the end of the pipeline and is currently responsible only for generating natural-language summaries from structured evidence.

The system can therefore run completely locally with no AWS account, no Bee account, and no LLM API key.

Challenges we ran into

Building around a personal baseline

One of the biggest challenges was avoiding the assumption that there is a universal "normal" way to speak.

Speech varies naturally between people, so comparing an individual directly against population-level thresholds can produce misleading results. We therefore designed a rolling personal baseline and explicitly handle the period where there is not enough historical data.

Handling missing information honestly

Real-world speech data is incomplete.

For example, some metrics require timestamps, audio, or sufficient text. Instead of estimating values when the necessary evidence is missing, TheraVoice marks them as unavailable and records why.

This graceful-degradation behavior became an important design principle throughout the system.

Keeping AI from becoming the decision-maker

Adding an LLM introduced another architectural challenge: how do we benefit from natural-language generation without allowing a generative model to determine clinical or medication-related outcomes?

We addressed this by keeping the core analysis deterministic. The LLM receives structured evidence and generates a summary, while event detection, medication handling, and therapy logic remain outside the generative model.

Integrating real-world data sources

We also needed TheraVoice to work both as a standalone research prototype and with real Bee data.

That led us to implement multiple Bee adapters:

MockBeeAdapter
      │
      ├── BeeSyncAdapter
      │
      └── BeeProxyAdapter

This allowed development and testing without requiring access to a physical device while still supporting real local Bee workflows.

Accomplishments that we're proud of

We're particularly proud that TheraVoice is more than an isolated biomarker prototype.

We built an end-to-end working system where:

  • A transcript or audio sample can enter through the API.
  • Biomarkers are extracted automatically.
  • A personal baseline is created and updated over time.
  • Deviations are detected against that baseline.
  • Context is incorporated into the analysis.
  • Multiple specialized agents process the available evidence.
  • Results are persisted into a longitudinal patient timeline.
  • A daily or clinical-oriented summary can be generated.
  • Bee conversations can be synchronized directly into the pipeline.
  • An optional LLM can produce a natural-language summary.
  • The entire workflow can run locally without external services.

We're also proud of the safety-oriented engineering decisions:

No fabricated biomarkers. No automatic medication changes. No diagnostic claims. No population baseline masquerading as personalization. No raw audio persistence by default. No LLM processing without explicit consent.

What we learned

Building TheraVoice taught us that developing an AI system for a sensitive domain is not only about making the model more sophisticated.

A useful system also needs to answer:

  • What evidence do we actually have?
  • What information is missing?
  • How confident should the system be about a change?
  • Is this observation meaningful for this particular person?
  • What should happen when an external AI service fails?
  • Which decisions should never be delegated to an LLM?
  • How do we preserve user consent and privacy throughout the pipeline?

We also learned the value of modular architecture. Separating ingestion, biomarkers, baselines, detection, context, agents, storage, and reporting makes the system easier to test, extend, and replace.

Most importantly, we learned that "I don't have enough evidence" is sometimes a better system response than producing a plausible-looking number.

What's next for TheraVoice

TheraVoice is still an evolving open-source project.

Our next steps include:

  • Richer longitudinal analysis

    • More robust personal-baseline modeling
    • Longer-term trends
    • Improved change detection

  • More advanced speech analysis

    • Additional acoustic and linguistic biomarkers
    • Better robustness to recording conditions
    • More multilingual analysis

  • Stronger agentic capabilities

    • More capable context-aware agents
    • External LLM providers beyond the current integrations
    • Better separation between deterministic evidence and generative reasoning

  • Improved Bee integration

    • More robust synchronization
    • Persistent change-feed state
    • Better handling of real-world data availability

  • Production security

    • Stronger authentication and authorization
    • Rate limiting
    • Account recovery and verification
    • More comprehensive deployment controls

  • Evaluation

    • Larger-scale longitudinal experiments
    • Robustness testing
    • Evaluation of false alerts and missing-data behavior
    • Human-centered evaluation of summaries and dashboard usability

Our long-term vision is to turn TheraVoice into a transparent, privacy-conscious, open-source framework for longitudinal communication monitoring — one that helps people understand changes in their own data while keeping clinical decisions where they belong: with people and their healthcare professionals.

Built With

Share this project:

Updates

Submission history