Step Inside - Live the story

A crew of AI agents reads your page, writes the scenes, paints the world, voices every character and films each moment, while OpenCV directs. Then you step inside the story in VR.

Comment

Inspiration

Every child who loves a story wants to be inside it. A picture book gives you one frozen picture per page; we wanted the page itself to open like a door: point a camera at any story, and a few minutes later you are standing in that world while the characters act it out around you. We set ourselves one hard rule: it must work for any page, a poem, a picture book, a screenshot, with nothing prepared in advance.

What it does

Step Inside turns one page into a living 360° story you watch line by line, in a browser or on a Meta Quest.

  • Scan, upload or film a page. A live OpenCV coach talks you through the shot ("move closer", "hold still", "tilt away from the light") and captures it automatically. A page it has seen before is recognized and replayed instantly.
  • A crew of AI agents builds the story in parallel. A story agent splits the page into moments, assigns every line to the narrator or a character, and decides what happens in each moment. Painters illustrate every moment, voice agents read each line in a different voice per character with word-by-word captions, a world agent paints a 360° world for every moment, and an animator films each moment as a short video in which the characters act.
  • OpenCV checks every agent's work before you see it: it restores lines the story agent skipped, rejects pictures with stray lettering, erases figures painted into empty worlds, cuts out the character the line is about, and inspects every video clip frame by frame. Clips that cut away, zoom, lose the painting or show no action are rejected and retried.
  • Then you step in. The story opens when it is completely made, with a progress screen showing each scene as it is filmed. Each moment plays on a curved, depth-displaced screen inside its own 360° world, whose sky drifts with the wind OpenCV read from the brushstrokes, with a glowing sun or moon, flickering lamps at night, haze and weather. Glide with the thumbstick for real parallax; Replay, Pause, Next and Exit sit in a small dock you can grab and move anywhere.

How we built it

OpenCV 5 does the seeing, checking and directing.

  • Reading: page flattening (contours, warpPerspective), PP-OCRv3 text detection and CRNN recognition through cv.dnn, run as parallel network copies (8.2 s → 1.9 s on a 160-word poster), Otsu polarity for light-on-dark text.
  • Active perception: the camera coach measures text size, Laplacian sharpness, glare and framing on every frame; ORB features + RANSAC homography recognize known pages; the sharpest frame is picked from an uploaded video.
  • Depth and life: Depth Anything V2 (cv.dnn) gives every moment and world its relief; colour, texture and depth masks find sky, water and leaves to animate; a structure tensor reads the wind direction from the sky's brushstrokes; blob analysis finds the sun, moon and lamps.
  • Moving characters: YOLOX (cv.dnn) on tiles plus depth pop-outs find the character the line names; a judge checks size, solidity and border contact; GrabCut, seeded by the detector box and filtered by depth, cuts it out; YuNet reads which way its face looks.
  • The clip critic: ORB + RANSAC fidelity to the painting, PP-OCRv3 lettering, YOLOX figure counts, Farneback optical flow on the moving pixels to confirm the described action and catch camera moves, SSIM and colour-histogram distance to detect cuts, and a loop-point search.

AWS runs the agents and the app. Amazon Bedrock: Nova Lite (story agent, streamed), Stability AI Stable Image Core (moments) and Inpaint (360° worlds, backgrounds behind moving characters), Luma AI Ray2 (image-to-video, queued one clip at a time and checked by OpenCV). Amazon Polly (neural voices with speech marks). Fargate on Graviton, S3, DynamoDB, Step Functions and CloudFront, all defined in AWS CDK.

Orchestration. A conductor runs the agents in parallel and streams every result to the viewer the moment it exists: painting starts while the story agent is still writing, voices while pictures paint, filming the instant a moment is painted. The viewer is three.js with WebXR.

Challenges we ran into

  • Making it move, not just look 3D. Our first version was beautiful stills in 360°. Real motion needed a video model, and the one AWS video model we planned around turned out to be retired on our account despite being listed as authorized. Only a real API call revealed it. We moved to Luma Ray2.
  • Video models don't follow instructions. Early clips cut to a different place mid-shot or zoomed in instead of animating anyone: 0 of 3 passed our critic. We fixed the causes rather than loosening the checks (close framing on one or two characters, locked-off camera wording, smarter checks). On the next story, The Hare and the Tortoise, all 6 of 6 moments became video.
  • Published books. The story agent was asked to copy the page word for word, and Bedrock's content filter blocked it on copyrighted pages. We built a "lines mode" in which the model only points at OpenCV's numbered lines and suggests spelling fixes, so the page is never reproduced.
  • Model output quirks. The story agent sometimes nested all its moments in the wrong place, which silently fell back to raw text; quoted speech without quotation marks went to the narrator. Each failure we saw on a real page became a fix and a test.

Accomplishments that we're proud of

  • Any page works, with nothing prepared in advance. A story is painted and voiced in about 8 seconds, and fully filmed in minutes.
  • OpenCV is not a preprocessing step but the director: it checks every agent and acts on what it sees (retry, repair, reject) in 11 closed loops.
  • The first fully filmed story: The Hare and the Tortoise, 6 of 6 moments as video, the first clip after 76 seconds, every accepted clip faithful to its painting.
  • It runs in a phone or desktop browser and on a Meta Quest, from one codebase.
  • An honest evaluation: every failure we hit is documented in the report with its cause and fix.

What we learned

  • Generative models are unreliable alone; a vision critic that checks their output turns them into something you can ship.
  • Streaming and parallelism matter more than raw model speed: a few seconds feels instant, a blank minute feels broken.

What's next for Step Inside - Live the story

  • Lip sync, as soon as a talking-character model is available on AWS; until then an OpenCV mouth puppet driven by Polly's viseme timings.
  • Character consistency across moments, so the same hero looks the same in every scene.
  • Whole books: chapters as connected worlds you walk between.
  • Classroom mode: a teacher scans a page and the whole class steps into it together.

Built With

Share this project:

Updates

Submission history