Spatially Aware Property Inspection

One phone video becomes a measured, photoreal 3D model of the room. OpenCV 5 finds and pins damage, shows what was never filmed, and an AI agent proposes accept, repair or refer.

Comment

Inspiration

Insurers and lenders now receive more and more of their property evidence as a phone video filmed by an owner, tenant or field agent, not an inspector's visit. We had built an AI property-inspection platform before, and two problems kept coming back:

  1. Coverage. People film what they notice. In our phone walkthroughs only 20–43% of the room surface was filmed well enough to inspect, and ceilings (where leaks show first) were almost never filmed. Nobody finds out until the video is reviewed, when the person has left.
  2. Evidence. A crack in a video frame has no position, no size and no context. It can't be found again, measured, or compared with the next inspection.

So we asked: what if one phone video could become a measured 3D model of the room, where every wall, floor and ceiling knows whether it was actually inspected, and every defect sits on the surface where it is?

What it does

From one phone walkthrough:

  • Hybrid visual SLAM and 3D reconstruction. Depth Anything 3 predicts depth and camera pose; OpenCV 5 checks and corrects them: LK optical flow, SIFT + MAGSAC pose verification, trajectory refinement, triangulated landmarks, point-to-plane ICP, a 1.5 cm TSDF mesh. The room is levelled to gravity, the walls snapped to clean planes and textured from the video.
  • Photoreal 3D. 3D Gaussian splats are trained on the engine's cameras, so you can walk through the room as it looks in the video, with every finding pinned in 3D.
  • Coverage you can trust. Every 10 cm of surface is checked against every frame, so the system knows what was seen, what was inspectable and what was inspected, and what to film again.
  • Defects, placed and measured. OpenCV finds candidates (cracks, stains, mould, rust, peeling paint), measures them in metres and places them on the room model. Anything that is really furniture, a door, a picture frame or a room seen through a doorway is ruled out with the reason.
  • Evidence integrity. Reused, spliced, looped or filmed-off-a-screen footage is flagged with perceptual hashes, histogram cuts and FFT moiré.
  • An agent that acts on what it sees. It looks again from better views, dismisses or flags candidates, asks for missing surfaces to be re-filmed, grades the property and proposes ACCEPT, REPAIR or REFER. A person approves or overrides; the agent never confirms damage on its own.

How we built it

Geometry first. Monocular depth has no scale, so we fit the floor with RANSAC and set metric scale from the camera height $h$ above it:

$$ s = \frac{h}{\operatorname{median}_k \left( \mathbf{n}^\top \mathbf{c}_k + d \right)} $$

where $\mathbf{c}_k$ are the camera centres and $(\mathbf{n}, d)$ the floor plane. OpenCV then verifies every frame pair with an essential matrix, refines the trajectory, and ICP aligns each depth map to the growing model before TSDF fusion.

Coverage as geometry. A surface sample $\mathbf{x}$ with normal $\mathbf{n}_x$ counts as inspectable in frame $k$ when

$$ \lVert \mathbf{c}_k - \mathbf{x} \rVert \le 3\,\text{m}, \qquad \angle!\left(\mathbf{n}_x,\, \mathbf{c}_k - \mathbf{x}\right) \le 70^\circ, \qquad z_k(\mathbf{x}) \le D_k!\left(\pi_k(\mathbf{x})\right) + \varepsilon, $$

and the frame is sharp: close, square-on, not hidden behind furniture. Keyframes are chosen by greedy set cover of the inspectable surface.

Defect or object? Each candidate's pixels are ray-cast onto the room planes and compared with depth. Something standing in front of the wall is furniture; something behind it is seen through an opening. Grounding DINO and MobileNetV2 (through cv2.dnn) then catch objects that sit flat on the wall, such as frames, clocks and switches.

Risk. A rules engine turns findings into a condition score, with each finding's risk points $p_i$ discounted by the number of views $v$ that saw it:

$$ \text{score} = 100 - \frac{\sum_i p_i}{\sqrt{v}} $$

That score drives the grade A–E, insurance perils, repair cost, valuation impact and the underwriting decision. Below 60% inspected the decision must be REFER.

Stack: OpenCV 5 (contrib), Depth Anything 3, Open3D, Grounding DINO, gsplat, FastAPI, three.js, Amazon Bedrock for the agent (with a rule-based policy over the same tools as a fallback).

Challenges we ran into

  • Furniture looks like damage. Brown chairs read as rust, wooden doors as water stains. Classical detectors alone were unusable; only room geometry plus semantics made them trustworthy.
  • Scale and short baselines. Monocular scale depends on the assumed camera height, and the translation direction between neighbouring frames is poorly constrained. Rotation agrees with OpenCV to under 1°, translation direction much less.
  • OpenCV 5 is new. We hit API changes (an empty projectPoints returning None) and an OpenCL path in cv2.Volume that failed on one driver.
  • A laptop is not a server. Running the reconstruction while installing GPU packages hard-locked the machine. We learned to cap every thread pool and run one heavy job at a time.
  • Gaussian splats on Windows. gsplat only ships prebuilt wheels for Python 3.10, so splats run in their own environment.

Accomplishments that we're proud of

Tested on two real homes, all on a laptop RTX 3080:

  • Geometry that holds up. ICP cut the depth disagreement between frames from 4.5 → 2.5 cm and 3.9 → 2.7 cm. Landmark reprojection median is 1.9–2.3 px, and OpenCV's refinement halved the trajectory jitter. The second room reconstructs as 8.2 × 8.4 m with a 3.11 m ceiling.
  • Photoreal rooms. On frames they never saw, the Gaussian splats reach 28.4 dB / 0.93 SSIM and 24.7 dB / 0.86, against 20.5 and 18.1 dB for the textured mesh. You can walk inside the room along the exact path of the video.
  • False alarms down from 60 to 1. On a room with no damage, the candidates reaching a person fell from 60 → 12 → 1 as we added room geometry and semantic checks.
  • Fraud checks that work. All four doctored test videos (reused, spliced, looped, filmed off a screen) were flagged; the honest ones raised no flags.
  • A full product, not a notebook. Upload with a live scanning progress view, a dashboard, a 3D viewer with numbered pins, finding review, human approval and an exportable report.

What we learned

  • Learned models are strongest when geometry checks them. Depth Anything 3 gives a great first guess, and OpenCV's epipolar checks and ICP make it measurable.
  • Being honest about coverage is the feature. "We didn't see the ceiling" is worth more to an insurer than a confident "no damage found".
  • Agents need hard limits: they propose, measure and ask, and a person decides.

What's next for Spatially Aware Property Inspection

  • Footage of rooms with real cracks, damp and mould to measure detection recall.
  • LiDAR / RoomPlan import from our iOS capture app, for exact scale.
  • A room-builder agent that turns the measured model into an editable, dimensioned floor plan and as-built 3D model.
  • A pay-per-use AWS deployment: a GPU batch job that starts per walkthrough and scales back to zero.

Built With

Share this project:

Updates

Submission history