Holograph

GMaps, for where the sun don't shine

Comment

Inspiration

Every large building in India is legally required to hang an Emergency Escape Route Plan on every floor — a printed map with room names, numbers, staircases, lifts and exits. There is one a few metres from wherever you're standing right now. Thousands of buildings, a uniform format, drawn to a consistent visual grammar.

And nobody uses them for anything except a fire.

That's the whole idea. The map problem for indoor spaces is usually framed as an acquisition problem — you need the venue to cooperate, hand over CAD files, and let you run a WiFi fingerprinting survey. But the building has already published its own floor plan, on the wall, in a standard format, at no cost and with nobody's permission required. We wanted to find out how far you could get from a photograph of a sign.

The second thing that pushed us: indoor navigation is where language models are most tempting and most dangerous. Asked "how do I get to the robotics lab", a model will happily invent a corridor, a distance, and a room that doesn't exist — confidently, in fluent prose. We wanted to build the version where that is structurally impossible.

What it does

Holograph turns fire-escape boards into a live, routable spatial twin of a building that you can talk to in plain English.

  • Ask for directions. "Take me to the robotics lab on the fifth floor, no stairs." It traces the route through a real graph and draws it on a 3D model of the building.
  • Step-free routing that actually differs. Ground floor to Computer Lab-4 is 71 m up the staircase, or 115 m via the east lift. Tick "step-free" and the drawn line visibly moves from the stair shaft to the lift shaft. It's a Dijkstra result over an edge attribute, not a cosmetic label.
  • Tell it what changed. "The grand staircase is shut for repairs." The space turns red and any open route re-plans around it, live.
  • Show it what changed. Upload a walkthrough video to /observe; it samples frames, asks the model which spaces are blocked, and mutates the graph accordingly.
  • Point a camera at a corridor and /locate picks which of the building's known spaces you're standing in — from a closed set, never an invented one.
  • Read the building as a stack. All seven floors at once, or any combination — each deck is an independent toggle, and a traced route stays drawn through the floors you've hidden.

The model we built covers KJ Somaiya's Bhaskaracharya building: 207 spaces across seven floors, from Machine Room and CNC Lab on the ground to the Bio Technology labs on the sixth — every room name and number read off the building's own boards. A second twin of Chhatrapati Shivaji Maharaj Terminus ships alongside it as a synthetic demo, to show the system isn't hardcoded to one building.

The rule underneath all of it: the language model never computes a route, a distance, or a coordinate. It translates a sentence into a command. Python and networkx do every piece of geometry. Every id the model returns is checked against the building graph before anything happens — an id that doesn't exist becomes unknown rather than reaching the screen.

How we built it

Deliberately small. Six files, about 1,900 lines, no database, no build step, no npm, no bundler, no deployment.

Backend — FastAPI + networkx. One building.json loaded into memory at startup becomes a graph. Routing is Dijkstra weighted by edge distance, with impassable edges excluded and — when you ask for step-free — inaccessible ones too. That single filter is what makes the stairs-versus-lift answer real. Route steps are generated from node types by template, never by a model.

The model layer — Gemini Robotics ER 2, four calls, all closed-set. locate, identify and observe take images; chat takes a sentence. Every one returns a Pydantic schema, and every id inside it is then validated against the world model. The chat endpoint returns a command the server has already executed — the reply's factual half is an f-string over the real routing result, so there's nothing left for the model to hallucinate.

Frontend — one HTML file, vanilla JS, inline SVG. The 3D view is a hand-written isometric projection: no three.js, no WebGL, no dependency at all. Volumes are extruded from each room's rectangle, depth-sorted, and painted deck by deck. A Fusion-style view cube in the corner shares the scene's exact rotation maths — 26 clickable orientations, where a corner click lands on true isometric because the pitch falls out of the geometry.

Reading the boards. We photographed all seven, then ran each through greyscale, autocontrast, 2× Lanczos upscale and an unsharp mask before tiling them into overlapping crops. That pipeline is what made the ground floor's small labels readable at all.

Challenges we ran into

The photographs fought back. These are signs behind glass in a corridor — glare, reflections, keystone distortion, 8pt type. Raw, half the ground floor was illegible. The enhancement pipeline wasn't a nicety; without it we'd have had three floors, not seven.

The 3D model was a Necker cube. Our first build drew every face with the identical flat tint, and the whole building became bistable — it flipped inside-out as you looked at it, and neither of us could agree which floor was on top. It was reported as "the hologram is confusing, it's an optical illusion", and the fix wasn't more detail, it was light: walls shaded by a fixed world-space sun, so the eye has something to anchor the solid.

Painter's algorithm doesn't care about your intentions. We sorted every polygon by centroid depth in one global pass. A floor deck is an enormous plane and a room is a small box standing on it, so comparing their centroids let deck tiles paint over the rooms sitting on them. The fix was to stop treating it as a sort problem: painting is grouped per deck — deck N, everything standing on deck N, then deck N+1 — because that order is physical fact, not a heuristic.

Our own tests lied to us. Half the HUD buttons were dead in the browser, and our test suite passed. The stage's pointerdown handler was calling setPointerCapture, stealing the pointer before any button could receive its click — but our test used JavaScript's .click(), which dispatches synthetically and bypasses pointer capture entirely. The test exercised a path no real user takes. Rewriting it to fire real pointerdown → pointerup → click sequences immediately turned red, and also caught a second bug: a drag whose pointerup never arrives (alt-tab mid-orbit) left the camera spinning forever.

A CSS rule with no owner. The route chart rendered stretched and blurry — but only on one team member's Mac. A bare svg { width:100%; height:100% } rule, written for the main stage, was applying to every SVG on the page. The stretch factor was the parent panel's height divided by the chart's, so it varied with window size and OS scrollbar behaviour: identical code, different distortion per machine.

The board and the building disagreed. The sixth-floor plan shows nine rooms in the teaching row. Standing in the building, there are five classrooms, then two washrooms about a quarter the width of a classroom, and the corner lab is 613 — in the west wing, where it always was. It took three rounds of correction from someone physically there to get that row right. Printed plans are a starting point, not ground truth.

Accomplishments that we're proud of

Seven photographs became a routable seven-storey building. 207 spaces, 238 connections, every pair reachable, no overlapping geometry — generated from a source anyone can walk up to and photograph.

Step-free routing that means something. Most accessibility toggles are a filter on a list. Ours changes the path: stairs edges are marked inaccessible, so asking for step-free forces the route through a lift and the answer genuinely differs — 71 m versus 115 m, and you can watch the line move.

A language model that cannot invent geometry. Not "we prompt it carefully" — the architecture doesn't give it the chance. It emits an intent; Python does the rest; every id is checked against the graph. We think this is the interesting contribution, more than any individual feature.

A 3D building viewer with zero 3D dependencies. Projection, depth sorting, lighting, picking and an interactive view cube, in plain SVG in one file that opens with no build step.

We were honest about what we don't know. The layout comes from the building; the distances are our estimates and the UI says so. We'd rather ship a caveat than a number we can't defend.

What we learned

Constraining a model beats trusting one. Validation guarantees the model can only name a room that exists — but it does not guarantee the room is the right one. A confidently wrong yet real id passes straight through. Knowing exactly where your guarantee stops is more useful than believing it covers more than it does.

Test at the layer the user actually touches. .click() passing while every button was dead is the lesson of the whole build. A green test on a synthetic path is worse than no test, because it buys false confidence.

Physical data needs a human in the loop. We read the plans carefully and still got the teaching row wrong three times. Somebody who walks that corridor daily corrected details no amount of image enhancement would have surfaced.

Depth is physics, not a sort key. Trying to order a scene with one clever comparator kept producing impossible pictures. Encoding the actual structure — floors stack, things stand on floors — made it correct by construction.

Reach for the smaller model. Our chat layer still runs on the robotics vision model because we reused the image endpoints' call path without revisiting it. It works, it's just the wrong tool for parsing a sentence — slower and dearer than it needs to be.

What's next for Holograph

Ranked by what actually unblocks the rest:

  1. Score the camera localization. The eval harness was built early and never got its photos, so /locate has no accuracy number. Everything else is speculation until it does — and it's the feature that makes "where am I" a single tap instead of a dropdown.
  2. Calibrate the distances. The graph is right; the scale is estimated. One tape-measure pass down a corridor per floor makes every number in the interface real.
  3. Put it on a phone. Today it's a desktop layout, which is backwards for a tool whose user is lost and holding a phone. Camera capture plus a mobile view is the honest form of this product.
  4. Extract the boards automatically. Right now a human transcribes each plan. Going from photograph to graph without that step is what turns one building into a campus.
  5. Persist and share state. Blockages live in memory and vanish on restart. Making an observation durable — and attributable — is what separates a demo from something a facilities team could actually run.

The long version: fire-escape boards are a legally mandated, visually uniform, machine-readable map of nearly every large building on earth, and they are sitting completely unused. Holograph is a bet that they're the cheapest indoor mapping dataset nobody has bothered to pick up.

Built With

Share this project:

Updates

Submission history