OpenSpeak

Inspiration

Most TTS systems for Indian languages are either locked behind expensive commercial APIs or require compute most builders don't have access to. We wanted to prove out a lightweight, from-scratch approach — small enough to train and run on a single consumer GPU — as a foundation for genuinely underserved-language TTS, starting with the core pipeline and scaling to Indian languages next.

What it does

OpenSpeak trains a compact 350M-parameter language model (LiquidAI's LFM2.5) to directly predict SNAC neural-codec audio tokens from text, then decodes those tokens into speech — the same core recipe used by larger systems like Orpheus-TTS, but built to run end-to-end on an 8GB consumer GPU instead of a cluster.

How we built it

• Expanded the base model's vocabulary with 28,672 new tokens representing SNAC's 3-level hierarchical audio codec, so the LM can "speak" in acoustic tokens the same way it speaks in text.
• Built a full training + inference pipeline tuned specifically for 8GB VRAM: gradient checkpointing, 8-bit AdamW, gradient accumulation.
• Validated the approach on English (LJSpeech) as a proof-of-concept for the pipeline, since it's the best-resourced dataset for iterating quickly — with the architecture designed to be dataset-agnostic for extending to Indian languages next.
• Built portable, Kaggle-compatible inference so training and testing can happen across different hardware.

Challenges we ran into

• Diagnosed a VRAM-spillover bottleneck silently capping training speed — GPU showing "100% utilization" while power draw sat at ~30W of a 123W ceiling, revealing it was memory-bound, not compute-bound.
• Caught a tokenizer/vocab consistency bug where copying checkpoint files without their training-time special tokens would have silently broken inference.
• Learned firsthand how much training an audio-token LM needs before output moves from structured noise to real speech — genuinely useful signal for scoping what a low-resource-language version will require.

Accomplishments that we're proud of

A working, end-to-end audio-token LM pipeline — vocabulary expansion, training, and portable inference — built and debugged from scratch on a single consumer GPU in under a week.

What we learned

How to budget GPU memory for training a language model with an expanded output vocabulary, how to diagnose training bottlenecks from GPU telemetry alone, and what loss ranges actually correspond to intelligible speech for this class of model.

What's next for OpenSpeak

Extend training to Indian-language speech datasets using the same architecture, continue training with LoRA to push past our current checkpoint's plateau, and support multiple voices/speakers via the existing per-speaker token design.

Built With

• python
• pytorch