DoodleTales

A magical storytelling adventure where your drawings come to life with Nova #AmazonNova

Comment

🎨 DoodleTales: The Magical Storytelling Scout (Amazon Nova Hackathon Report)

🌲 Inspiration

Every child's drawing is a window into a vast, hidden world. However, once the drawing is finished, it often remains static on a piece of paper or a digital screen. We were inspired by the idea of "waking up" these drawings—giving the characters voices, personalities, and the ability to go on new adventures while staying true to the child's unique artistic vision. We wanted to turn the act of drawing into a collaborative, living dialogue between a child and an AI companion. The launch of the Amazon Nova 2 family of models provided the perfect multimodal toolkit to turn this dream into a reality.

⚙️ What it does

DoodleTales is an interactive AI agent that brings children's art to life through three core pillars:

  1. Vision & Recognition: Using Amazon Nova 2 Pro, the app identifies and segments individual characters and objects within a drawing, mapping their precise locations with "magical halos."
  2. Multimodal Adventure: A voice-enabled "Storytelling Scout" engages the child in a conversation. The Scout asks about the characters' names, ages, and roles, reacting dynamically to the child's voice and the drawing itself using Amazon Nova 2 Pro's multimodal capabilities.
  3. Style-Consistent Storytelling: As the adventure unfolds, the AI generates new scenes for the story. Crucially, these new images use Amazon Nova 2 Canvas with the original drawing as a "Style Reference," ensuring that the characters, colors, and "hand-drawn" aesthetic are perfectly preserved in every new frame.

🧱 How we built it

Architecture: https://github.com/StnkRB/Nova-DoodleTales/blob/main/ARCHITECTURE.md

DoodleTales is a full-stack application built with a focus on visual fidelity and creative consistency:

  • Frontend: Built with React 19 and Tailwind CSS 4, using Motion for fluid UI transitions. We used a custom canvas layer to render the "magical halos" based on AI-detected coordinates.
  • AI Intelligence (Amazon Bedrock):
    • amazon.nova-2-pro-v1:0: Powers the high-precision character segmentation and the conversational "Storytelling Scout." It handles both the visual analysis of the drawing and the complex reasoning required for interactive storytelling.
    • amazon.nova-canvas-v1:0: Generates new story scenes. We leverage the "Style Reference" feature to ensure the AI respects the child's original artistic style.
  • Backend: A Node.js/Express server acts as a secure proxy to AWS Bedrock, managing credentials and orchestrating the flow between the frontend and the AI models.
  • Tool Integration: We implemented custom tool specifications (highlight_character, generate_story_image) that allow the Nova 2 models to interact directly with the frontend state, creating a seamless "agentic" experience.

🧗 Challenges we ran into

  • Style Preservation: One of the biggest hurdles was preventing the AI from "fixing" the child's drawing. Modern image generators often default to professional 3D or digital art styles. We had to carefully tune the conditioning parameters in Amazon Nova Canvas to lock in the "crayon and paper" aesthetic.

  • Coordinate Mapping: Vision models return coordinates in a normalized 0-1000 scale. Mapping these accurately to a responsive, object-fit image container in React required building a custom coordinate transformation hook that accounts for aspect ratio differences.

  • Multimodal Context Management: Keeping the "Storytelling Scout" aware of both the visual state of the drawing and the ongoing conversation history required a robust state management system to ensure the AI didn't "forget" what a character looked like or what its name was.

🏅 Accomplishments that we're proud of

  • Character Persistence: We achieved a level of character consistency that feels truly magical. When a child draws a specific character, that character appears in every subsequent scene of the story, looking exactly as the child intended.
  • The "Magic Halo" Loop: The seamless interaction where the AI identifies a character, and a golden glow immediately appears around it on the screen. It creates an instant emotional connection between the child and the AI.
  • Seamless Migration: Successfully migrating the entire project from a different AI ecosystem to Amazon Nova 2 in record time, proving the versatility and developer-friendliness of the Bedrock API.

📚 What we learned

  • The Power of Conditioning: We learned that Amazon Nova Canvas is incredibly powerful when given a strong style reference. It doesn't just copy the image; it understands the vibe of the art.
  • Agentic UX Design: We discovered that the most engaging AI experiences aren't just chat boxes—they are interfaces that the AI can "reach out and touch" through tool calls.
  • Nova 2 Pro's Vision Capabilities: We were impressed by how well Nova 2 Pro handles the "messy" and non-standard geometry of children's drawings, which often baffle traditional computer vision models.

🔭 What's next for DoodleTales

  • Amazon Polly Integration: We plan to replace the browser's default text-to-speech with high-fidelity, expressive voices from Amazon Polly to give the "Storytelling Scout" more personality.
  • Digital Storybook Export: Allowing parents to save the entire adventure as a narrated digital book that can be shared with family and friends.
  • Collaborative Worlds: Enabling multiple children to upload drawings that the AI then "merges" into a single shared universe where characters from different drawings can meet and interact.
  • Interactive Props: Letting the child draw new items (like a magic wand or a treasure chest) during the conversation and having the AI immediately recognize and incorporate them into the story.

Built With

Share this project:

Updates