visoto

Point your camera at your desk. Ten seconds later you're jumping across it, and the game already proved you can actually win before it let you play.

Comment

Visoto: Project Story

Inspiration

Most "AI makes a game" demos work the same way: point a model at a photo, get back some rectangles, make all of them solid platforms. You get a level built from your photo, but not a level built from your room — every object behaves identically, so the photo is just wallpaper behind generic gameplay.

We wanted the physics to come from what an object actually is. A mug of coffee shouldn't be a platform, it should be a hazard, because it's ceramic full of scalding liquid. A cushion shouldn't be solid, it should be bouncy, because foam compresses and springs back. That's the whole idea in one sentence: semantic physics — reasoning from what something is, not just where its edges are.

Once we had that idea, the actual hard problem revealed itself immediately: a vision model returns plausible boxes, not a playable level. Boxes overlap, float in mid-air, or leave the goal stranded across a gap nothing can jump. Ship that straight to a player and you get an app that looks broken about a third of the time. So the real project became: how do you turn "the model's best guess" into "a level we can mathematically prove someone can finish" — and then keep that promise honest as you add monsters, hazards, secret passages, and everything else that makes a level actually fun.

How we built it

The vision call is deliberately small. One file, app/api/analyze/route.ts, is the only place that knows a model provider exists. It sends a photo to Gemini and gets back structured JSON — bounding boxes, a material for each object, and a one-sentence reason for that classification. We wrote the schema in Gemini's own native spatial convention (box_2d: [ymin, xmin, ymax, xmax], points as [y, x], normalized to 0–1000) instead of asking the model to translate into our engine's coordinate system — that one change measurably improved box quality, because we stopped asking the model to do arithmetic in its head before answering.

The solver is the real engine. Everything the model returns passes through a pipeline before a player ever sees it:

  1. Sanitize — clamp boxes, drop frame-sized ones, remove duplicates
  2. Ground — guarantee a floor so the player can't spawn into the void
  3. Graph — turn each solid's top edge into a node
  4. Prove — breadth-first search across that graph using the actual jump arc from the physics engine
  5. Repair — insert bridges or stepping stones until the goal is reachable

Step 4 is the heart of it. A player's jump is a projectile: constant horizontal speed, gravity pulling down. For a launch velocity $v_0$ and a landing point that's risen by height $h$ above takeoff, the flight time is

$$t = \frac{v_0 + \sqrt{v_0^2 - 2gh}}{g}$$

and the horizontal distance covered is $x = v_{\text{move}} \cdot t$. That formula lives in exactly one place and is imported into the solver rather than copied, because if the solver's idea of a jump ever drifted from the physics engine's, the "proof" would be describing a player that doesn't exist.

Everything else is procedural, on purpose. Six playable characters, drawn on canvas, no sprite sheets. Sound effects synthesized from oscillators at runtime, no audio files. A coin economy where every individual coin pays out exactly once, ever, so restarting a level to farm it earns nothing. A WebGL renderer that extrudes every detected object out of the photo into a lit, shadowed 3D diorama — but strictly as a view over the same 2D simulation, because the completability proof only makes sense on a 2D plane, and we weren't going to risk that guarantee days before a deadline.

We also wrote eight ADRs (architecture decision records) documenting the load-bearing choices as we made them — including the ones that turned out wrong before we fixed them, because that's the more useful record.

Challenges we ran into

The proof kept being technically correct and practically wrong. Reachability math alone wasn't enough, and we only found out because we also built a second, independent check: a bot harness (npm run verify) that actually plays hundreds of randomized attempts on the real physics engine and checks whether any of them finish. It caught things the math missed one at a time:

  • A jump arc only checks where you land, never what's directly overhead — a lamp above the route could wedge a player permanently.
  • The repair step itself could build a trap: bridges stacked 63px apart for a 46px player, sealing the player into a shaft.
  • A goal floating above an empty floor could be "proven reachable" by erecting a tower straight up out of nothing, technically satisfying the math while being an absurd answer.

The scariest bug was the one that made things worse before it made them better. Our reachability check only ever looked at distance and height between two surfaces — it never asked whether the arc between them passed through fire. Fixing that one function (canTraverse) immediately broke more levels than it fixed, because the platform-placement code that builds bridges and rescues stranded goals was still hazard-blind, and kept proposing routes the now-stricter check correctly rejected, with nowhere to fall back to. We ended up having to thread hazard-awareness through the entire reachability graph and add one final, unconditional guarantee that re-verifies the whole level no matter which earlier repair step left it broken.

Real photos exposed things synthetic tests never would. A user uploaded a photo of their desk, and a bonus-room pipe rendered under a shelf with 30 pixels of clearance for a 46-pixel-tall player — solid, visible, and completely unusable. Our tests had verified every step of the climb and missed the destination. We rewrote the check to ask "does every surface actually have room to stand on it," not just "is the route we had in mind possible," and it immediately caught a second, identical bug we didn't know was there.

And the unglamorous stuff mattered just as much. A model we picked as a fallback turned out to have been quietly retired and returned 404 for new API keys, which cascaded into "levels sometimes fail to generate for no visible reason." A timeout budget that gave each model in our fallback chain its own allowance meant the client could give up while the server was still successfully working — so a slow-but-fine request looked identical to a broken one. Neither of those is exciting to fix, but both were the difference between a demo that works and one that mysteriously doesn't.

What we learned

The interesting problem was never "can a model find objects in a photo." It can. The interesting problem is everything downstream of that: turning a plausible guess into a guarantee, and being honest — in the UI, in the code, in the commit history — about exactly how much repair work that guarantee actually costs.

Built With

Share this project:

Updates