VoiceToLegacy

Inspiration

So many family stories, from the Holocaust generation to ours, live only in memory. Writing an autobiography can feel overwhelming, especially for older adults who have incredible stories to share but may be uncomfortable with technology or intimidated by a blank page. We created VoiceToLegacy to replace that blank page with a patient conversation and help preserve memories that might otherwise be lost.

What it does

VoiceToLegacy is an AI-powered oral-history interviewer. A storyteller speaks naturally while the AI listens, asks thoughtful follow-up questions, and creates a live transcript.

Interview & Edit: Select one or more conversations and transform them into an editable, evidence-grounded autobiographical chapter.
Organize & Trace: Manage interviews, edit generated chapters, and trace statements back to their source transcripts.
Export: Export the finished story as a branded PDF.

How we built it

We built VoiceToLegacy with vanilla JavaScript and a Node.js 22 server using only built-in Node modules.

Voice & Transcription: The OpenAI Realtime API powers the live voice conversation and transcription through WebRTC.
Guidance & Generation: The OpenAI Responses API provides contextual interview guidance, extracts structured claims from transcripts, and turns those claims into polished first-person chapters.
Storage & Validation: Data is stored as local JSON files. Our generation pipeline uses strict schemas, sentence-to-evidence mappings, grounding validation, and an automated audit to reduce unsupported details.
Resilience: We added resumable evidence extraction, automatic saving, bounded retries, accessible editing controls, and local PDF generation.

Challenges we ran into

Conversational Flow: Creating a natural voice interview was more difficult than simply connecting speech to an AI model. We had to carefully manage pauses, interruptions, finalized transcriptions, and the timing of follow-up questions so the interviewer felt patient rather than robotic.
Factual Accuracy: Memoirs are personal; an invented name, date, emotion, or detail can undermine the entire experience. We built an evidence ledger that connects generated sentences to specific transcript turns and prevents drafts with unresolved grounding problems from being treated as final.
Long-running Durability: API timeouts or temporary failures should not discard completed work. To solve this, we introduced checkpointed extraction, atomic local writes, controlled retries, and recoverable drafts.

Accomplishments that we're proud of

A Complete Journey: We are proud that someone can speak about their life, receive meaningful follow-up questions, save the interview, generate a chapter, edit it, and export it as a PDF.
Built-in Trust: VoiceToLegacy doesn't store your microphone audio, keeps your data in local project files, keeps the API key on the server, preserves source provenance, and flags chapters that need review.
Technical Rigor: The application has zero runtime package dependencies, and its offline test suite currently passes all 172 tests across the interview, storage, generation, security, and editing workflows.

What we learned

Timing is Everything: Good voice experiences depend as much on timing and restraint as they do on model intelligence. A strong interviewer asks one clear question, gives the storyteller room to think, and follows the emotional thread of the conversation.
Trust Requires Structure: Trustworthy AI writing requires more than a good prompt. It requires structured evidence, provenance, validation, failure recovery, and honest communication when the system is uncertain.
Empowering the Storyteller: Most importantly, we learned that technology can make storytelling more accessible without replacing the storyteller’s authentic voice.

What's next for VoiceToLegacy

Next, we want to expand from individual chapters into complete, organized autobiographical books. We plan to add photographs and personal documents, life-event timelines, more writing styles, multilingual interviews, and tools for family members to collaborate on stories.

The publishing possibilities are virtually limitless. Future capabilities could include:

Audio Options: Recording the storyteller narrating their own book in their natural voice.
Publishing Integrations: Integrating with Amazon KDP for print and eBook publishing, as well as Audible/ACX for producing, marketing, and selling audiobook editions.
Physical Keepsakes: Producing professionally designed hardcover books that families can preserve for generations.
Legacy Expansion: Expanding into additional publishing formats and personalized legacy experiences.