SCENE DIRECT

Inspiration AI video is astonishing and impossible to direct. You prompt, you pray, you re-roll. Filmmakers don't work that way — they block a scene, pose actors, and move a camera. We wanted the missing middle: take any photo and turn it into a 3D scene you can actually direct, then render the shot you staged. Photo → previz → film. What it does Drop in a photo. Scene Direct reconstructs the people as posable rigged actors — standing where they stood, facing what they faced — and rebuilds the set as editable 3D objects. Pose joints, move furniture, set keyframes, record a camera move, then render the take into AI video. A full product around it: projects, credits, Stripe billing, tutorial, and a companion phone app for walking the camera through the scene. How we built it Two AI systems working different jobs. GPT-5.6 (Luna) is the runtime brain: a director reads the photo and writes a scene manifest in real-world meters (sizes, anchors, facing conventions, matched sets), a coder swarm writes literal three.js for each object in parallel with the photo attached, and a critic reviews rendered rounds against the photo with typed, executable fixes. Codex was the parallel engineer: it built the staging pipeline (protected deploys + live Playwright smoke on every push), the production photo→scene flow, request hardening, and test coverage — in its own branches and PRs, receipts in the git history. Instruments do the measuring: SAM 3D Body keypoints, metric depth (Depth Anything 3, raw arrays), and raycast contact facts. 576 tests green. What we learned The week's biggest lesson became our architecture: instruments measure, solvers execute, critique judges. Vision models are superb at meaning ("that's a booth, it's missing its twin") and unreliable at arithmetic — and some truths are structurally invisible to vision: identical mannequins make a mirrored arrangement look exactly as correct as the real one, and a 2D render can't distinguish sitting-on from clipping-into. So those became measured laws code enforces — photo ordering can't swap, measured facing can't flip, seating is verified by raycast — while the model spends its intelligence on what only it can know. Challenges we ran into Coordinate frames above all: SAM's keypoints are per-person local, monocular depth compresses distance between facing people, and every provider speaks a different frame. We solved it by making the camera a fixed datum and promoting every hard-won geometric fact from "prompt guidance" to code-enforced law. The other challenge was auditability — we ended up logging every prompt and image every model call receives, so every failure came pre-diagnosed. What's next The critique loop (already proven in the repo's scene-lab) moving fully in-app, SAM object capture for hero-prop fidelity, and the phone-as-camera workflow polished for release.

Built With

• nextjs
• supabase