StoryFactory

Tell a bedtime story using your own voice

Comment

Inspiration

Bedtime stories are often remembered not only because of the story itself, but because of the familiar voice telling it. We wanted to preserve that feeling while making personalized audio creation easier. A parent might want to record stories for a child when they are away. A couple might want to create a comforting voice message. A content creator might need a reusable narration voice without repeatedly recording every script. Most existing AI voice tools require uploading sensitive voice samples to the cloud. StoryFactory was inspired by a different idea: what if users could create personalized voice content while keeping their voice data entirely on their own device?

What it does

StoryFactory is a local-first AI voice content studio for parents, couples, and content creators. Users can: Record or upload an authorized voice sample and create a reusable voice profile. Ask AI to generate a personalized script from a topic, tone, audience, and duration. Write or paste their own script without sending it to a remote language model. Review and edit the complete script before generating any audio. Generate narration locally using Qwen3-TTS and the selected voice profile. Regenerate an individual segment or the entire recording. Apply adjustable DeepFilterNet3 speech enhancement without rerunning TTS. Play the finished work as one continuous audio track. Save, favorite, and export the final result as WAV or MP3. StoryFactory requires no registration or cloud media storage. Voice samples, generated audio, projects, and playback data remain on the local machine. Only the text fields required for AI script generation are sent to the configured language model.

How we built it

We built StoryFactory as a local single-user web application optimized for Apple Silicon. The frontend uses React, Vite, TypeScript, Tailwind CSS, TanStack Query, Zustand, the MediaRecorder API, and the Web Audio API. It provides voice recording, script editing, generation progress, project management, and a continuous audio player. The backend uses FastAPI, Pydantic, SQLAlchemy, Alembic, and SQLite in WAL mode. Long-running audio generation is handled through persistent jobs so interrupted tasks can recover after the application restarts. For content generation, StoryFactory calls deepseek-v4-pro through an OpenAI-compatible API. Model responses are required to follow a structured JSON format and are validated before they can be saved or passed into the audio pipeline. For speech generation, we run Qwen/Qwen3-TTS-12Hz-1.7B-Base locally through oMLX. Each script segment is generated and stored independently, allowing failed segments to be retried without regenerating the entire project. After generation, the raw segments are joined into a 48 kHz mono narration master. DeepFilterNet3 processes the complete track to reduce noise while preserving speech continuity. FFmpeg then mixes the original and enhanced audio according to the selected strength, normalizes loudness, and produces a continuous 24 kHz master for playback and export.

Challenges we ran into

One of the biggest challenges was making independently generated TTS segments sound like one continuous recording. Playing separate files introduced audible pauses and decoding gaps. Processing every segment independently also caused differences in noise, loudness, and voice texture. We solved this by preserving the individual segments for retry and editing, while assembling a separate project-level master for playback. DeepFilterNet3 processes the entire joined narration instead of restarting for every segment, and FFmpeg handles pauses, fades, loudness normalization, and final encoding. Noise reduction was another challenge. Traditional high-pass and FFT filters could reduce stable background noise, but they could not reliably distinguish speech from TTS artifacts. Strong filtering also made lower voices sound thin. Replacing that approach with DeepFilterNet3 produced better speech-aware enhancement, while a dry/wet control lets users decide how much of the enhanced signal to use. We also had to design around unreliable local model services, long inference times, invalid model responses, interrupted jobs, and partial failures. That led us to use persistent job states, bounded retries, atomic file writes, strict audio validation, and recovery logic that preserves completed segments and previous valid masters.

Accomplishments that we're proud of

We are proud that StoryFactory is more than a voice-cloning demo—it is a complete local content workflow. The application supports voice capture, AI or manual writing, script review, local synthesis, segment-level regeneration, full-track speech enhancement, continuous playback, project storage, and WAV/MP3 export. We are especially proud of the privacy boundary. Reference recordings and generated speech never need to leave the user’s machine, and manual scripts can remain completely local. We also built the audio pipeline to be resilient. Completed segments survive later failures, generation jobs can recover after a restart, and changing the enhancement strength can remaster existing audio without spending time regenerating speech. If enhancement fails, the last valid master is preserved.

What we learned

We learned that generating speech is only one part of creating a convincing audio experience. Segmentation, pauses, loudness consistency, noise characteristics, file validation, playback behavior, and failure recovery are just as important as the TTS model itself. We also learned that privacy needs to be an architectural decision rather than a feature added later. Separating remote text generation from local voice processing made the system easier to reason about and reduced the risk of sensitive audio being sent to the wrong service. Finally, we learned that speech enhancement should be applied at the complete-track level. Running enhancement independently on each segment can introduce new boundaries, while processing the assembled narration gives the model continuous context and produces a more consistent result.

What's next for StoryFactory

Next, we want to make StoryFactory easier to install and useful across more creative workflows. Our planned improvements include: Packaging StoryFactory as a native desktop application. Supporting additional operating systems and local inference backends. Adding optional background music with independent narration and music controls. Providing cover-art generation and richer project presentation. Adding more detailed controls for speech enhancement, low-frequency rumble, and mastering. Improving automatic evaluation of noise, clipping, loudness, and voice consistency. Supporting reusable content templates without introducing predefined voice roles. Exploring real-time conversational companionship while preserving the local-first privacy model. Adding encrypted backup and export options for voice profiles and projects. Our long-term goal is to make StoryFactory a private, general-purpose studio where anyone can turn meaningful words and an authorized familiar voice into audio worth keeping.

Built With

Share this project:

Updates