PlaceCast

PlaceCast- from one photo to a map-ready building, with every door and window found, oriented and verified.

Comment

Inspiration

What it does

Inspiration

Procedura AI is building Scorched Nebraska, a game set on real-world maps. They asked for buildings generated from a photo and placed on the map, with one thing that matters most for gameplay: doors must be separable from windows and face the right way, so players can interact with them. We built PlaceCast to do exactly that and to know when it can't.

What it does

Take one phone photo of a building plus its address:

Find the building. Geocode the address, pull the real footprint and height from OpenStreetMap, and place it on a live Cesium globe at the correct position, scale and rotation. Find every opening. Detect doors, windows, entrances and garage doors in the photo, tell them apart, and reject look-alikes: lamps, columns, balconies, recessed entryways. Put them on the real building. Rebuild the phone's camera from its GPS, compass and focal length, refine it against the building's outline in the photo, and cast each detection onto the real footprint's walls. Every opening becomes its own object with a type, compass bearing, size in metres, confidence, and a decision: ACCEPT, or REVIEW with the reason. Restyle it for Scorched Nebraska (rusted, plated, scorched), textured onto the building from the same camera.

Click any door on the map and you get its type, bearing, size and why it was accepted or flagged.

How we built it

We used pretrained models for image editing (FLUX Kontext), 3D generation (TRELLIS), segmentation (SAM 2) and open-vocabulary detection (Grounding DINO). What we built: the geometry pipeline (footprint conditioning, oriented bounding boxes, Umeyama alignment, IoU refinement), the EXIF camera model and ray casting, the door/window classification logic, and the confidence gate, a model we trained and evaluated on our own benchmark of 1,680 placements (synthetic corruptions of real footprints, scored against known truth). The stack is Python, PyTorch, trimesh, FastAPI and CesiumJS.

Challenges we ran into Single-photo 3D generation gets depth wrong. Our own gate rejected all six AI-generated meshes. The New Classroom Building came back 1.6:1 when the real building is 2.2:1. So we place the real footprint and put the openings on it, instead of trusting the mesh. Doors vs windows is hard. Adding "distractor" words (lamp, column) to the detector's prompt halved its window scores, so we split it into two prompts to keep scores calibrated. Focal length and camera distance are indistinguishable from a silhouette, so we fixed focal from EXIF and searched only position and pitch. Symmetric buildings: all 12 wrong auto-accepts in our benchmark were one mirrored, symmetric building (Cassell Coliseum). Geometry alone can't catch that; only the photo can. Accomplishments that we're proud of The detector called Burruss Hall's bay window an entrance. The 3D step caught it, because its bottom sits 1.1 m above the ground, and a door has to reach the ground. A door hidden behind a lamp post is flagged "partly occluded", not lost. On three of six buildings, the camera fit failed our own check, so the system refused to place those openings instead of guessing. A door on the wrong wall is worse than no door. What we learned

Measure everything. Coverage isn't accuracy: Microsoft's building heights covered almost every building but were 44–67% too low on this campus. "Roads" near campus buildings were almost all footpaths. And a system that says "I don't know" with a reason is more useful than one that is confidently wrong.

What's next

Multiple photos per building to fix depth and verify more cameras, modelling porches and towers the footprint misses, and exporting openings as native interaction sockets for the game engine.

Built With

Share this project:

Updates

Submission history