StoryForge

Inspiration

We wanted to create an interactive storytelling experience that combines creativity, artificial intelligence, and technology to make reading more engaging for children. Traditional storybooks have a fixed storyline, but we wanted children to become active participants by creating their own characters and deciding what happens next. This inspired us to build StoryForge, a picture-book maker that transforms children's imagination and drawings into personalized adventures.

What it does

StoryForge allows children to draw their own hero or type an idea to begin a personalized story. A locally running language model generates the story one page at a time, allowing children to influence the plot through interactive choices or their own ideas.

Each page features an AI-generated illustrated background, the child's character animated on top of the scene, and audio narration. Children can tap difficult words to see and hear their definitions, making the experience both entertaining and educational. The application supports different reading levels, from pre-readers to confident readers.

How we built it

We built StoryForge around the NVIDIA Jetson Orin Nano Developer Kit, using a locally hosted AI model to generate stories while integrating cloud services for additional creative features.

AI storytelling: We used Qwen 2.5 3B Instruct, running locally through llama.cpp, to generate structured story outlines and individual pages. Character creation: We developed a drawing interface that allows children to draw their own heroes. Image processing separates the character from the background so it can be animated within the story. AI-generated illustrations: We integrated FLUX.1-schnell through Hugging Face to generate illustrated backgrounds, with local image-generation and preset-image fallbacks. Narration: We integrated ElevenLabs for audio narration, with local Piper and browser speech synthesis as fallback options. Reading assistance: We used word-frequency analysis and readability tools to identify difficult vocabulary and generate child-friendly definitions. Animations: We used HTML Canvas, CSS animations, SVG filters, and JavaScript to bring characters and story scenes to life. Ambient lighting: We connected an ESP32 running Arduino code with FastLED to control a WS2812B LED strip that changes colors to match the illustrations. Backend: We used Python, FastAPI, Uvicorn, Pydantic, and asynchronous processing to coordinate the AI models, illustrations, narration, and story progression.

Challenges we ran into

One of our biggest challenges was running AI models on the Jetson Orin Nano's limited shared memory while maintaining a responsive experience. Generating stories, illustrations, and narration requires several components to work together without making children wait too long between pages.

Another challenge was ensuring that AI-generated stories remained appropriate, coherent, and consistent. Smaller language models can produce malformed responses, repeat information, or generate unexpected content. To address this, we used structured JSON outputs and Python validation to check, repair, and filter generated responses before displaying them.

We also had to coordinate multiple services with different response times. To improve responsiveness, we started generating page backgrounds as soon as the scene description became available instead of waiting for the entire page to finish. We implemented fallbacks for image generation and narration so the application could continue working when external services were unavailable.

Finally, animating a child's drawing over generated backgrounds required image processing to separate the character from the original background and make the character fit naturally into different scenes.

Accomplishments that we're proud of

We are proud of building an interactive storytelling application that combines locally hosted AI with cloud-based creative services on an NVIDIA Jetson Orin Nano.

Our biggest accomplishment is bringing multiple technologies together into one cohesive experience. Children can create a character, generate a personalized story, explore illustrated scenes, listen to narration, learn new vocabulary, and influence the plot through their decisions.

We are also proud of implementing safeguards and fallback mechanisms. Rather than displaying raw model output, StoryForge validates structured responses and checks story content before presenting it to children. Generating illustrations and narration asynchronously also helps reduce waiting time.

What we learned

We learned how to integrate locally hosted language models with cloud-based AI services and hardware peripherals into a single application. Working with the Jetson Orin Nano taught us to consider memory constraints, model quantization, and performance when designing AI-powered experiences for edge devices.

We also learned the importance of structured outputs, input validation, and fallback mechanisms when building applications around language models. AI-generated content cannot always be trusted to follow instructions perfectly, so robust validation is essential, especially when developing for children.

What's next for StoryForge

We would like to expand StoryForge with more drawing tools, richer character animations, additional story choices, and more interactive elements. We also want to improve the speed and consistency of locally generated illustrations to reduce reliance on cloud services.

Future improvements could include more personalized reading recommendations, progress tracking, additional languages, and expanded educational features. We would also like to enhance the physical experience with more responsive lighting effects and explore other ways to connect digital stories to the real world.

Built With

• arduino
• css3
• elevenlabs
• esp32
• flux
• html5
• hugging-face
• javascript
• jetson-nano
• llama.cpp
• ngrok
• python
• qwen