SpatialMind

SpatialMind turns a phone walkthrough into a private PC-hosted 3D room twin, with local scene understanding, object detection, and an interactive viewer.

Comment

Inspiration

I kept running into 3D room scans that looked cool but told you nothing. You get a mesh, you spin it around, and... that's it. No sense of whether it's actually accurate, no way to build on it. I wanted something a homeowner, student, designer, or renter could actually use — walk through a room with your phone, and end up with a private digital twin you can inspect and trust, not just admire.

That gap between reconstruction and understanding is what SpatialMind is built around.

What it does

You record a walkthrough on your phone, send it over to a PC, and the PC does the heavy lifting — the original video and full 3D assets stay local, and the phone stays lightweight.

Under the hood, SpatialMind:

  • selects clear, well-spread frames from the video instead of using every frame;
  • boosts contrast just enough to help camera matching (not for looks);
  • tracks camera movement and checks whether there's actually enough parallax to reconstruct anything in 3D;
  • reconstructs dense geometry with COLMAP;
  • builds a browser-based 3D viewer with orbit, zoom, top/front views, wireframe mode, and export options;
  • runs local object detection on common room items;
  • generates a scene summary, schematic floor plan, and report — but only when the capture evidence actually supports it;
  • keeps track of which frames COLMAP actually registered, instead of assuming every scan worked.

For the strongest run so far, SpatialMind registered all 33 selected camera views and produced a dense point cloud of 400,000+ observed points, plus a surface mesh.

How we built it

  • Frontend: Next.js, React, TypeScript, Tailwind CSS, Three.js (with OrbitControls)
  • Backend: FastAPI, Python
  • Vision pipeline: OpenCV for decoding, sharp-frame selection, overlap checks, and frame prep
  • Reconstruction: COLMAP for camera tracking, dense depth estimation, and meshing
  • Asset packaging: Trimesh to build and validate GLB, PLY, and OBJ outputs
  • Object detection: YOLO, run locally
  • Assistant: Ollama running a local Qwen model, so the optional in-app assistant works without a paid API key

I used Codex with GPT-5.6 throughout the build — planning milestones, architecting the Next.js/FastAPI split, debugging the mobile-to-PC upload flow, tuning COLMAP settings, improving the viewer, writing tests, and keeping the README current.

The harder problem was making the reconstruction honest. It's easy to build something that always shows you a result — the hard part is teaching it to say when a result isn't trustworthy. Codex helped me add frame registration diagnostics, stationary-camera checks, camera-baseline checks, bridge-frame selection, dense-point filtering, and mesh refinement, all aimed at preventing SpatialMind from presenting a weak or partial capture as a reliable room model.

Challenges we ran into

Photogrammetry in ordinary rooms is genuinely hard. Blank walls, motion blur, close-ups, a camera that barely moves, gaps in coverage — any one of these can quietly produce sparse or misleading geometry, and a naive pipeline will hand you a confident-looking result anyway.

The biggest lesson here was that showing uncertainty matters as much as generating the model. Getting the upload flow working reliably between phone and PC, and tuning COLMAP so it neither over-promised nor silently failed, took a lot of iteration.

Accomplishments that we're proud of

  • Built an end-to-end pipeline that goes from a phone-recorded video to a validated, browser-viewable 3D reconstruction
  • Added real reliability checks — parallax validation, registration tracking, baseline checks — so the app can tell the difference between a good scan and a bad one instead of faking confidence
  • Hit 33/33 registered camera views and 400,000+ points on our best reconstruction
  • Kept the whole thing runnable without any paid API key, including the in-app assistant
  • Made a clear, explicit distinction in the product between a surface mesh, a point-cloud preview, and "this scan needs to be redone" — the app never invents walls, furniture, or measurements it didn't actually see

What we learned

That showing uncertainty is a feature, not a limitation. It would've been easy to always output something and let it look impressive; it was much harder — and much more valuable — to build a system that knows when its own output can't be trusted yet.

What's next for SpatialMind

The next direction is semantic 3D localization: tying detected objects to their exact positions in the reconstructed scene. That opens the door to stronger floor plans, layout recommendations, accessibility planning, and simulation-ready room models — all while keeping a hard line between what was actually observed and what's being estimated.

Built With

Share this project:

Updates