Inspiration
Everyone carries a camera that knows where it is in space. Almost nobody can use it to measure their own bedroom.
Apple solved room scanning for the small slice of people holding a Pro iPhone or an iPad Pro, because RoomPlan needs the LiDAR sensor. The phone in most pockets is shut out of the entire category.
The second thing mattered more to us. Even with the sensor, what comes back is a mesh: a few hundred thousand triangles with no idea what a wall is. You can look at it. You cannot ask it how wide the room is, you cannot move the desk, and you cannot hand it to anything that expects a floor plan. The sensor was never the real limitation. The representation was.
So we went after both at once. Run on any ARKit iPhone, and produce something structured enough to actually be useful afterwards.
What it does
GhostMap turns one physical room into semantic geometry: four ordered floor corners, walls derived from those corners, doors and windows as parametric openings, and furniture as typed objects with measured dimensions.
Instead of walking into all four corners and tapping once at each, you trace the line where each wall meets the floor. Every frame casts a ray, so a single wall arrives as hundreds of measured points rather than one. The corners are then computed where adjacent walls cross, so you never have to reach one, including the corner behind the sofa that the old flow simply could not capture. Then you take ceiling height, mark the doors and windows, and point the phone at your furniture, which measures itself.
The phone streams the room to a desktop viewer as you go. Once the scan finishes, the desktop takes ownership: orbit the room, select a piece of furniture, change its dimensions by hand, measure between any two points, save and reload. Every object exports as its own glTF file, sized from the real object, ready to drop into Blender or a game engine.
A finished room is a few kilobytes of JSON. Not tens of megabytes of triangles, and every number in it is yours to edit.
How we built it
Two Unity projects, one shared package. apps/scanner is the iOS capture app on AR Foundation and ARKit, apps/viewer is the desktop reconstruction and editing app, and com.ghostmap.shared owns every domain model, every geometry utility, every validation rule and the wire format. Neither app is allowed to reimplement a rule the shared package already states.
Plane detection runs exactly once. The floor lock gives us an origin, an up axis and a forward direction, and from that moment the hardware is never asked for anything else. Every later measurement is the centre-screen camera ray crossed with a plane whose equation we already know. Ray-plane intersection is arithmetic, not sensing, which is the entire reason this works without a depth sensor.
Walls are fitted, corners are computed. Holding the sweep control while tracing a wall's floor junction casts a ray every frame, so one wall becomes a cloud of up to 2048 floor points lying along a line. Four corner taps gave us four single-sample measurements with nothing to average and no way to tell a good one from a bad one. A traced wall gives us a fitted line and a residual that reveals whether the aim wandered. We fit by total least squares, accumulating a $2{\times}2$ scatter matrix about the centroid:
$$ S = \begin{bmatrix} S_{xx} & S_{xz} \ S_{xz} & S_{zz} \end{bmatrix}, \qquad S_{xx} = \sum_i (x_i - \bar{x})^2, \quad S_{xz} = \sum_i (x_i - \bar{x})(z_i - \bar{z}) $$
with eigenvalues
$$ \lambda_{\pm} = \frac{(S_{xx} + S_{zz}) \pm \sqrt{(S_{xx} - S_{zz})^2 + 4S_{xz}^2}}{2} $$
The eigenvector of $\lambda_{+}$ is the wall direction, and $\sqrt{\lambda_{-}/n}$ is the RMS perpendicular residual, which gates whether a sweep is accepted. Corner $i$ is then the intersection of wall $i-1$ and wall $i$.
Furniture measures itself. The whole method is one observation: a horizontal surface at height $y$ is the top of something $y$ tall. The height of a desk is not estimated, it is the height we found the surface at. Width and depth come from the surface's own extent, and its yaw comes from the surface's own axis, which is why a desk sitting at an angle to the wall comes out at that angle rather than snapped square to the room.
The link between the apps is plain TCP on the local network, newline-delimited JSON, and no cloud at any point. The phone sends the entire room after every structural change rather than sending deltas. That sounds wasteful and it is exactly why there are no sync bugs: there is no event ordering left to get wrong. Ownership of the room moves from phone to desktop at a single instant, when the scan is finalized, so the two can never both think they own it. That one decision eliminates the whole class of distributed-state bugs this kind of app usually drowns in.
The glTF exporter is hand-written, straight to the binary container, and it builds its geometry by calling the same function the viewer's renderer calls. What you download cannot drift from what you saw on screen.
Challenges we ran into
Ordinary least squares is singular, and we caught it before it shipped. The obvious way to fit a wall is $z = mx + c$, minimising vertical residuals. That blows up for any wall running parallel to the $x$ axis, where $m \to \infty$, and in a rectangular room two of your four walls are in exactly that situation. Total least squares minimises perpendicular distance instead:
$$ \min \sum_i \left( (x_i - \bar{x})\sin\theta - (z_i - \bar{z})\cos\theta \right)^2 $$
No preferred direction, no singular case, at the cost of trading a closed form for an eigenvalue problem. Worth it.
Catastrophic cancellation nearly ate our residual. $\lambda_{-}$ is a difference of two similar large numbers. Accumulate the scatter matrix in 32-bit floats and the small eigenvalue, which is precisely what the quality check depends on, degrades into rounding error. It accumulates in double. Choosing the eigenvector needs a conditioning test between two candidate formulas, because each degenerates for a different wall orientation: one vanishes for a wall parallel to $x$, the other for a wall parallel to $z$. We take whichever has the larger magnitude.
We characterised our own error model exactly. Aiming at the wall-floor junction from eye height $h$, at horizontal distance $d$, with angular aim error $\delta$, the ground error is
$$ \varepsilon \approx \delta \left( h + \frac{d^2}{h} \right) $$
Quadratic in distance, and that shaped the interaction directly. Because the error grows with how far away you are aiming, the right way to capture a wall is to trace it from close up and move along it, rather than standing back and covering it in one sweep from across the room. The maths told us what the gesture should be.
We spotted that we had deleted our own smoke alarm. When corners were walked to, we got an error check for free: measure four corners, see whether the loop closes. Once corners became the intersections of fitted lines, they close exactly, by construction, for any scan at all. The check silently became a tautology. Catching that a passing test had stopped meaning anything, rather than enjoying the green tick, is the kind of thing that only turns up if you are actually reading your own system.
A clean merge is not a correct merge. Two feature branches each widened the same constructor, in separate new files. Git reported no conflict. Our test suite caught it immediately on the first compile, which is exactly what 855 tests are for: conflict markers find textual disagreement and are blind to semantic disagreement.
Unity is left-handed and glTF is right-handed, and we learned that the way everyone does, with every surface rendering inside out. The exporter now authors faces directly in glTF space with the winding derived from the normal, so it cannot happen again.
Accomplishments that we're proud of
855 tests, all green, covering the geometry, the validation rules, the protocol and both apps. When the full integration branch met a compiler for the first time, it came up with a single real defect.
The eigen decomposition was right first time. Thirty-two tests on the wall fitting alone, and all thirty-two passed on first execution, including the conditioning branch that chooses between the two eigenvector formulas.
Tracing changed the product, not just the code. Walking into four corners is awkward in a furnished room, and a corner behind a sofa or a radiator cannot be aimed at all, so some rooms were simply uncapturable. Tracing a line works wherever that line is visible, and it turns one tap per corner into hundreds of samples per wall.
Furniture that measures itself. No model, no classifier, no inference cost, no network. A flat surface at 74 cm is the top of something 74 cm tall, and the hardest number to get is the one you get for free.
A system that stays honest with itself. Walls are derived from corners on demand and never stored, so a room cannot hold corners and walls that disagree. Both apps validate against the same shared package. Every significant decision has an ADR recording what we chose and what we traded away.
What we learned
The representation is the feature. We started out thinking the hard part was getting 3D without a depth sensor. The hard part was deciding what to produce. A mesh records what a room looked like. Semantic geometry records what the room is, and only the second one lets anyone ask questions or make changes afterwards.
Let the error model design the interaction. Deriving $\varepsilon \approx \delta(h + d^2/h)$ told us more than any single accuracy number could. It said error grows with aiming distance, which is why the capture gesture is tracing a nearby line rather than covering a wall from across the room.
Make the seams where the swaps will happen. Our protocol reader takes a plain Stream, so nothing above the socket knows a socket exists. Swapping the transport later costs us almost nothing. The places we left a seam are the places that stayed cheap to change.
Constraints are a design tool. Refusing to use LiDAR forced us to think about what a room actually is rather than how to capture more points of it. The best part of this project came out of the thing we were not allowed to use.
What's next for GhostMap
Type suggestion with YOLOX, in progress now. A model pass over the camera frame suggests what each detected surface is, so furniture arrives already named. The dimensions keep coming from the measured geometry, and classification stays separate from measurement by design.
Drag and drop furnishing. Rearrange a scanned room, or furnish an empty one from a palette, straight in the viewer.
Transport with no network at all. Apple's Network framework carries a peer-to-peer connection over Bluetooth or direct Wi-Fi with no infrastructure network present. Since our protocol only wants a byte stream, this is a transport swap rather than a protocol change.
Rooms that are not boxes. Four corners today, any convex footprint next, and after that whole flats, with rooms joined at their doorways into a single coordinate frame.
Ask the room questions. Because the room is corners and measured boxes, "will this sofa fit through that door" is arithmetic. Deterministic, instant, and incapable of hallucinating. That is the payoff for having built structure instead of a scan, and it is where this goes next.
Log in or sign up for Devpost to join the conversation.