Personalized Voicemail Generator

Inspiration I wanted to explore what's possible when you combine modern generative audio with fan engagement. The idea: what if you could send a personalized voicemail message to fans in a beloved artist's actual voice—not a passable imitation, but something that genuinely sounds like them?

The spark came from seeing how far voice synthesis had come, yet how most demos still sounded robotic or unconvincing. I wanted to build something that felt real.

The Architecture Challenge My first instinct was to use a single voice synthesis model. But that approach had a hard ceiling—TTS-only models are optimized for intelligibility and speed, not voice fidelity. They can approximate a voice, but they can't nail the nuances that make someone sound like themselves.

That's when I realized: use two models, each for what it's best at.

ChatterBox handles fluent speech generation from text. Seed-VC handles precise voice conversion. By chaining them, I got expressiveness + authenticity. ChatterBox generates natural prosody and emotional delivery; Seed-VC locks it into the target voice with diffusion-based refinement.

What I Learned

1. Diffusion steps are load-bearing. More steps = better quality, but exponential time cost. Finding the sweet spot (30 for first pass, 50+ for refinement) was critical.

2. Reference audio quality matters enormously. A longer, cleaner reference clip teaches the model far more about the target voice than a short clip. The difference is night and day.

3. Text normalization isn't trivial. Hard newlines in the input cause ChatterBox to treat each line as a separate utterance—you get audible "cuts" between sentences. Collapsing whitespace solved it.

4. Mood presets need balance. Lower exaggeration with more diffusion steps creates natural delivery. High exaggeration with few steps sounds theatrical and rushed.

Building the Pipeline Fan Voice (optional) ↓ Analyze emotion/energy [Higgs Audio Understanding] ↓ Write personalized script [GPT-4 + Hanni persona] ↓ Synthesize with ChatterBox [Eigen AI, cloud TTS] ↓ Refine with Seed-VC [Local diffusion-based VC] ↓ Polished voicemail

Each stage is a separate module. This modularity let me iterate on parameters independently—tuning ChatterBox without touching Seed-VC, or swapping reference clips.

The Challenges Challenge 1: Hallucination past 45 seconds ChatterBox generates garbled audio and noise once you pass ~42 seconds of input. No documented reason; the model just breaks down. Solution: hard cap scripts at 150 words and trim intelligently to sentence boundaries.

Challenge 2: Reference length vs. fidelity Seed-VC internally caps references at 25 seconds, but the system supports longer clips. I created a fallback: prefer a 60-second reference if available, fall back to the short profile otherwise. More data = richer voice model.

Challenge 3: Prosody consistency Early versions had jarring pitch/tone shifts. The auto_f0_adjust flag in Seed-VC fixed this—it automatically shifts the synthesized pitch to match the target speaker's median F0, preserving their characteristic voice range.

Challenge 4: The speed/quality tradeoff Initial settings (diffusion_steps=10–15 for ChatterBox, =30 for Seed-VC) were fast but produced unconvincing voice cloning. I tuned these up—knowing it would cost 2–3× generation time but yield dramatically better fidelity. For a demo about voice quality, speed had to be sacrificed.

Reflection The biggest lesson: voice cloning is as much engineering as ML. The models are powerful, but orchestrating them—reference preparation, parameter tuning, safety nets—is where the magic happens. You need to understand not just what the models do, but why they break and how to work around it.

The two-stage pipeline works because it respects the strengths and weaknesses of each component. ChatterBox excels at fluent, expressive speech. Seed-VC excels at voice matching. Together, they produce something neither could alone.

That's the insight I'll carry forward: sometimes the best architecture isn't the fanciest model—it's the right combination of tools, each doing what it does best.

Built With

• eigen