Reclaim.

Restoring agency and identity to non-verbal individuals.

Comment

Reclaim

Your signs. Your words. Your voice.

Inspiration

Most AAC (augmentative and alternative communication) tools give voiceless users a voice — a generic, robotic one that sounds nothing like them. If you lost your speech tomorrow, a text-to-speech app would let you order coffee, but it wouldn't sound like you ordering coffee. Your slang, your phrasing, your actual voice — all gone.

We wanted to build the opposite: an app where you sign a phrase at your camera and the room hears it in your own words and your own voice. Not "the user requests a beverage" — but "yo can I get a coffee," in the voice you had.

What it does

Point your camera at yourself and sign. Reclaim:

  1. Translates the signing — Gemini ingests the recorded clip as native video and produces text.
  2. Lets you review it — nothing is ever spoken without your edit and approval.
  3. Rewrites it in your style — an LLM restyles the sentence the way you text, using a style profile distilled from your real messages.
  4. Speaks it in your voice — an ElevenLabs instant clone of your voice, chosen from your personal voice library.

Holding up 1–5 fingers at the start of a clip fires a quick phrase ("I want a coffee", "I'm tired, I'm going to head home", …) — deterministic shortcuts for the moments fluent recognition isn't enough. There's also a conversation mode: your partner speaks, Gemini transcribes them, and the app drafts three tappable replies in your style.

How we built it

The app is Next.js 16 (App Router) on DigitalOcean App Platform, with DO services doing most of the heavy lifting:

  • Style rewriting — Llama 3.3 70B on DO Gradient serverless inference. Each user has a style library of named profiles ("Casual", "Business", …), each with its own message corpus, embeddings, and an LLM-distilled style card.
  • Semantic few-shot retrieval — the corpus is embedded with GTE-Large on DO serverless embeddings; at speak time, the most similar past messages become the few-shot examples in the prompt.
  • TTS caching — DO Managed Valkey caches audio keyed on (voice, sentence): repeated phrases play instantly and never burn cloning credits twice. Qwen3 TTS on DO Gradient is the neutral-voice fallback, then device speech.
  • Storage — DO Spaces with a private per-user prefix; PII (URLs, emails, phone numbers) is redacted from the corpus before it's ever stored.
  • Fine-tuning tier — personal style runs on a three-tier ladder: retrieved examples → style card → a per-user LoRA fine-tune of Qwen2.5-1.5B (TRL) trained on the user's message corpus on an AWS EC2 GPU instance, pushed to Hugging Face, and served from a dedicated inference endpoint as that user's own model.

Challenges we ran into

  • Fluent ASL recognition is genuinely unsolved. Rather than pretend otherwise, we designed around it: every translation is editable before it's spoken, and the finger-count quick phrases give a deterministic path for high-frequency needs. Honesty about the model's limits became a design principle.
  • Voice cloning is biometric data. Voiceprints fall under GDPR Article 9, Illinois BIPA, and Texas CUBI. We made consent a hard gate — no upload or cloning happens until an explicit written consent record is stored — and "Delete my data" erases everything: the Spaces data, the consent record, and every clone at the provider.
  • Making one sentence sound like a specific person. A single prompt tweak wasn't enough. Getting there took the full ladder — semantic retrieval so examples match the sentence being restyled, a distilled style card so the model has an explicit description of how the user writes, and finally per-user fine-tuning when prompting alone stops being enough.
  • Latency budgets. Sign → speech has to feel conversational. The Valkey cache, serverless inference, and Flash-tier TTS each shaved seconds off the round trip.

What we learned

  • Designing around a model's weaknesses (editable output, deterministic shortcuts) beats waiting for the model to be perfect.
  • Style is layered: retrieval, distillation, and fine-tuning each capture something the others miss.
  • Consent and erasure flows are easier to build first than to bolt on — biometric data leaves no room for "we'll add it later."

What's next

First-party voice cloning: services/f5-tts/ already holds a self-hosted, MIT-licensed F5-TTS deployment on a DO GPU droplet, so voice biometrics never have to leave our infrastructure at scale. And wiring the per-user fine-tune trigger into the app UI, so the top tier of the style ladder is one tap away.

Built With

Share this project:

Updates