TTS

InspirationWe noticed a stark reality in our daily lives: visually impaired individuals and the elderly heavily rely on screen readers that sound incredibly robotic, lifeless, and often mispronounce words. This mechanical interaction creates a frustrating barrier to learning, working, and enjoying digital content. We believe technology should be human-centric. Our core inspiration was to build a community-driven, natural-sounding Text-to-Speech (TTS) engine that brings warmth, emotion, and accurate intonation to synthetic voice, completely free for accessibility tools.What it doesOur TTS software acts as a highly realistic, emotive voice synthesizer. It takes standard text input, normalizes it (handling numbers, abbreviations, and dates), converts it into phonetic representations, and generates high-fidelity audio.Unlike older concatenative or basic parametric TTS systems, our project produces smooth, continuous speech with natural breathing pauses and dynamic pitch control. It features a lightweight API that developers can seamlessly plug into screen readers, audiobook generators, and educational platforms.How we built itWe adopted an end-to-end deep learning architecture combining an Acoustic Model and a high-fidelity Vocoder.Text Processing Pipeline: We built a custom Grapheme-to-Phoneme (G2P) module tailored for tonal accuracy, heavily relying on a linguistic dictionary.Acoustic Modeling: We utilized an architecture similar to VITS (Variational Inference with adversarial learning for end-to-end Text-to-Speech), which directly predicts the audio waveform from text representations, bypassing the traditional two-stage Mel-spectrogram generation bottleneck.Adversarial Training: To achieve high-fidelity audio, our generative model is trained using a composite loss function that balances adversarial realism with structural accuracy. The total loss for the generator is defined as:$$\mathcal{L}G = \mathcal{L}{adv}(G) + \lambda_{fm} \mathcal{L}{fm}(G) + \lambda{mel} \mathcal{L}{mel}(G)$$Here, $\mathcal{L}{adv}(G)$ is the adversarial loss that pushes the generator to fool the discriminator, $\mathcal{L}{fm}(G)$ is the feature-matching loss to align hidden structures, and $\mathcal{L}{mel}(G)$ ensures the generated audio matches the target Mel-spectrogram. The weights $\lambda_{fm}$ and $\lambda_{mel}$ balance these objectives during backpropagation.Challenges we ran intoTonal Nuances: Modeling a tonal language (where a change in pitch alters the word's meaning entirely) was mathematically punishing. Minor errors in predicting fundamental frequency (F0) resulted in words sounding "warped" or grammatically incorrect.Data Scarcity: Deep learning thrives on data, but finding dozens of hours of high-quality, studio-recorded, single-speaker data with consistent emotion was difficult. We spent weeks manually cleaning audio data to remove background noise and correct transcripts.Compute Constraints: Training end-to-end generative audio models requires massive VRAM. We had to implement gradient checkpointing and mixed-precision training (FP16) just to get the model to fit on consumer-grade GPUs.Accomplishments that we're proud ofLow Latency: We optimized the inference pipeline to achieve a Real-Time Factor (RTF) of under 0.15 on a standard CPU. This means it takes less than 150 milliseconds to generate a full second of audio, making it viable for live screen-reading.Open-Source Impact: Releasing the core models and dataset back to the community to empower other developers building accessibility tools.User Validation: The most rewarding moment was testing the alpha build with visually impaired students and hearing their relief at finally listening to a voice that sounded like a real human reading to them.What we learnedAudio is Unforgiving: In computer vision, a slightly blurry pixel is often unnoticeable. In audio processing, even a single mismatched frame results in a sharp, painful "click" or artifact in the user's ear. Data quality is absolute.Empathy in Evaluation: Standard metrics like Mean Opinion Score (MOS) don't tell the whole story. We learned to evaluate our model based on "listener fatigue"—measuring how comfortable the voice is to listen to for hours on end, rather than just isolated sentences.What's next for TTSThe journey doesn't stop here. We are actively working on:Edge Deployment: Quantizing the model using ONNX so it can run entirely offline on mobile phones, ensuring users have access even without internet connectivity.Zero-Shot Voice Cloning: Allowing users to create a personalized digital voice using just 10 seconds of reference audio—particularly useful for patients losing their voice to conditions like ALS.Emotional Control: Adding latent slider controls so users can adjust the tone (e.g., joyful, serious, whispering) depending on the context of the text being read.

Built With

• python