lumen3d

Inspiration

3D reconstruction has become remarkably powerful, but many language-aware 3D systems still feel difficult to use outside a research environment. They often require calibrated cameras, lengthy optimization for every new scene, expensive GPUs, or complicated setup steps.

I wanted to explore a simpler question: What if someone could record a room with an ordinary phone, turn it into a 3D scene, and then search that scene using natural language?

That idea became Lumen3D. My goal was to make language-queryable 3D scenes accessible without requiring specialized cameras, per-scene model training, or a workstation-class GPU.

What it does

Lumen3D converts a casual video walkthrough, or a collection of photos, into an interactive 3D point-cloud scene.

Once the scene has been processed, a user can type a query such as:

“the red backpack”
“a wooden chair”
“the trash can”
“the object beside the door”

Lumen3D searches the objects found in the scene and highlights the closest match directly in the browser-based 3D viewer.

The system uses open-vocabulary search, so it is not limited to a fixed list of object classes. The scene only needs to be built once. After that, searches are lightweight enough to run on a CPU.

How I built it

Lumen3D combines three pretrained models, with each model handling a different part of the problem:

Depth Anything 3 reconstructs depth, camera poses, and camera intrinsics from selected video frames.
SAM 2 generates object masks for the visible objects in each frame.
SigLIP converts object crops and text queries into embeddings that can be compared in a shared image-text space.

The pipeline begins by sampling useful frames from the input video. Depth Anything 3 predicts the geometry of each frame, and I use NumPy-based unprojection to transform the depth pixels into a shared world-space point cloud.

SAM 2 then identifies possible objects in every frame. Each detected object is projected into 3D and embedded using SigLIP.

A major part of the system is determining when detections from different frames represent the same physical object. Lumen3D combines two signals:

Geometric similarity, based on overlap between the detections in a voxel grid
Semantic similarity, based on the cosine similarity of their SigLIP embeddings

A pair of detections is merged only when both signals agree. The final scene contains one embedding and one set of 3D points for each object instance.

I also built a command-line interface, an interactive browser viewer, a Colab workflow for users without local GPUs, automated tests, and a hosted demo. The build stage performs the computationally expensive reconstruction and segmentation, while the query stage loads only the completed scene and the text encoder.

Challenges I faced

Associating objects across different viewpoints

SAM 2 detects objects independently in every frame. This meant that the same chair could initially appear as several separate objects when viewed from different angles.

Using geometry alone was not reliable because nearby objects could overlap in 3D. In an early version, the system incorrectly merged a wooden door with a trash can beside it. Using semantics alone also caused problems because two similar chairs in different parts of a room could be merged.

Requiring both geometric overlap and semantic similarity produced much more reliable object instances.

Handling scale ambiguity

The geometry generated by feed-forward reconstruction models is internally consistent, but it is not necessarily measured in real-world meters. This made fixed distance and voxel thresholds unreliable.

I solved this by calculating thresholds relative to the reconstructed scene’s bounding-box diagonal. This allows the association logic to work across scenes of different sizes and scales.

Understanding the embedding scores

SigLIP similarity scores did not behave like traditional classification probabilities. Correct matches could still have relatively small raw scores, and unrelated images often shared a high baseline similarity.

Instead of using a fixed confidence threshold, I designed the search system to rank objects by similarity. This produced more stable results across different queries and scenes.

Keeping the demo affordable

Reconstruction requires GPU-heavy dependencies, but querying a completed scene should not. I separated the project into a build-heavy stage and a query-light stage.

The completed scene is stored as a point cloud and a small table of object embeddings. Searching it only requires encoding the user’s text and performing cosine similarity, allowing the hosted viewer to run on a free CPU environment.

What I learned

Building Lumen3D taught me that combining foundation models is not simply a matter of connecting their outputs. Most of the engineering work happens between the models: transforming coordinate systems, handling scale, filtering masks, grouping detections, managing memory, and designing reliable similarity logic.

I gained a much deeper understanding of multi-view geometry, depth unprojection, point clouds, vision-language embeddings, instance association, GPU memory constraints, Python packaging, and interactive 3D visualization.

I also learned the importance of designing around deployment limitations from the beginning. Separating scene construction from scene querying made the system easier to distribute, test, and demonstrate.

Accomplishments I am proud of

I am especially proud that Lumen3D is a complete end-to-end project rather than only an experimental notebook.

It currently includes:

Video and image-based scene reconstruction
Open-vocabulary object search
Multi-view 3D object association
An installable Python package and CLI
An interactive browser-based 3D viewer
A free Colab workflow
A hosted CPU demo
Tests for geometry and dependency isolation

Most importantly, it demonstrates that a language-queryable 3D scene can be created from an ordinary phone video without per-scene training or specialized camera equipment.

What’s next

The next steps are to improve object association in crowded scenes, reduce noisy or duplicate masks, support larger environments, improve mobile capture guidance, and make the viewer more intuitive.

I would also like to explore spatial relationship queries, such as “the bag under the table,” and eventually allow users to ask broader questions about the reconstructed environment rather than searching for only one object at a time.

Inspiration

3D reconstruction has advanced rapidly, but many language-aware 3D systems are still difficult to use outside a research environment. They often require calibrated cameras, specialized hardware, complicated setup, or expensive optimization for every new scene.

We wanted to explore a simpler idea:

What if someone could record a room with an ordinary phone, turn it into an interactive 3D scene, and then search that scene using natural language?

That idea became Lumen3D.

Our goal was to make searchable 3D environments more accessible. A user should be able to capture a casual walkthrough, reconstruct the space, and search for objects without manually labeling the scene, training a model for that environment, or owning a workstation-class GPU.

What it does

Lumen3D turns a phone video, or a collection of photos, into an interactive and language-queryable 3D scene.

After processing the recording, users can explore the reconstructed point cloud in a browser and search for objects using everyday phrases such as:

"the red backpack"
"a wooden chair"
"the trash can"
"the object beside the door"

Lumen3D compares the text query with the objects discovered in the scene, ranks the most relevant matches, and highlights the selected object directly in 3D.

The search is open vocabulary, which means the system is not restricted to a predefined list of object classes. The scene only needs to be reconstructed once. After that, searches are lightweight enough to run on a CPU.

How we built it

Lumen3D combines three pretrained foundation models, with each model solving a different part of the problem:

Depth Anything 3 reconstructs depth maps, camera poses, and camera intrinsics from selected video frames.
SAM 2 generates masks for objects visible in each frame.
SigLIP converts object crops and text queries into vectors within a shared image-text embedding space.

The pipeline begins by extracting useful frames from an uploaded video. Depth Anything 3 estimates the geometry and camera information for those frames.

We then use NumPy-based depth unprojection to convert image pixels into 3D world-space points. The reconstructed points from multiple camera views are combined into a shared point cloud.

SAM 2 processes each selected frame and proposes masks for the objects it can see. Every mask is projected into 3D, producing a small point-cloud fragment representing that detection.

The corresponding image crop is passed through SigLIP to generate a semantic embedding. This embedding describes what the object looks like in a way that can later be compared with natural-language queries.

Because the same physical object may appear in several frames, we built an instance-association system that combines:

Geometric similarity, calculated using overlap between detections in a voxel grid
Semantic similarity, calculated using cosine similarity between SigLIP embeddings

Two detections are merged only when both their geometry and semantics agree. The final scene contains one combined point cloud and one averaged embedding for each object instance.

When a user enters a search query, the text is encoded with SigLIP and compared against the stored object embeddings. The closest matches are ranked, and the selected object's points are highlighted inside the Three.js viewer.

We also separated the application into two stages:

A GPU-assisted reconstruction stage that runs the larger vision models
A lightweight query stage that only loads the completed scene and SigLIP text encoder

This design allows users to build their scenes using a free Google Colab GPU and later search them on an ordinary CPU.

Challenges we ran into

Associating the same object across multiple views

SAM 2 detects objects independently in each frame. As a result, the same chair viewed from six different angles initially appears as six unrelated detections.

Using geometry alone was not reliable because objects located close together could occupy overlapping regions. In an earlier version, the system incorrectly combined a wooden door with the trash can beside it.

Semantic similarity alone was also insufficient because two visually similar chairs on opposite sides of a room could be mistaken for the same object.

We solved this by requiring both geometric overlap and semantic similarity before merging detections. This significantly improved the quality of the final object instances.

Handling unknown scene scale

The reconstructed geometry is internally consistent, but it is not necessarily measured in real-world meters. A fixed voxel size or distance threshold might work for one scene and completely fail for another.

To address this, we calculate important geometric thresholds relative to the diagonal of the reconstructed scene's bounding box. This makes the association process adapt to differently sized scenes.

Interpreting vision-language similarity scores

SigLIP scores do not behave like traditional classification probabilities. Correct matches can still produce relatively small raw values, while unrelated objects may share a high baseline similarity.

A fixed confidence threshold therefore produced inconsistent results. Instead, Lumen3D ranks all object instances according to their relative similarity to the query.

Managing GPU memory

Running reconstruction, segmentation, and embedding models across many high-resolution frames can quickly exceed the memory available on free or consumer GPUs.

We added frame sampling and image downscaling controls so users can balance reconstruction detail, processing time, and available GPU memory.

Keeping the hosted demo affordable

The reconstruction pipeline requires GPU-heavy dependencies, but requiring a GPU for every text query would make the project expensive and difficult to deploy.

We separated the heavy build dependencies from the lightweight query server. The hosted demo loads a completed point cloud and object-embedding table, allowing it to run on a free CPU environment.

Accomplishments that we're proud of

We are proud that Lumen3D is a complete end-to-end project rather than only a research notebook or isolated model demonstration.

The project currently includes:

Video and image-based 3D reconstruction
Open-vocabulary natural-language object search
Multi-view object-instance association
Adaptive geometric thresholds for scenes with unknown scale
An installable Python package
A command-line interface for reconstruction, querying, and viewing
An interactive Three.js point-cloud viewer
A FastAPI-based query server
A Google Colab workflow for users without a local GPU
A hosted CPU demo on Hugging Face Spaces
Automated tests for geometry and dependency isolation
Docker support for reproducible deployment

We are especially proud that Lumen3D does not require per-scene model training, manually calibrated cameras, or specialized capture equipment.

A user can record an environment using an ordinary phone, build the scene on a free Colab GPU, and then search it using natural language on a standard CPU.

What we learned

Building Lumen3D showed us that combining foundation models involves much more than connecting one model's output to another model's input.

A large part of the work happened between the models. We had to manage coordinate systems, camera transforms, depth unprojection, mask filtering, scene scale, duplicate detections, GPU memory, embedding behavior, and browser visualization.

We developed a much stronger understanding of:

Multi-view geometry and camera transformations
Depth-based point-cloud reconstruction
Image segmentation and mask processing
Vision-language embedding spaces
Cosine-similarity-based retrieval
Voxel-based 3D object association
GPU memory and dependency management
Python package and CLI development
FastAPI deployment
Interactive 3D visualization with Three.js

We also learned the value of separating expensive preprocessing from inexpensive inference. By building the scene once and storing a compact object-embedding table, we made repeated searches much faster and easier to deploy.

What's next for Lumen3D

The next steps are to improve object association in crowded scenes, reduce noisy or duplicate masks, support larger environments, improve mobile capture guidance, and make the viewer more intuitive.

We also want to improve the visual quality of reconstructed scenes and give users clearer feedback about which camera angles or areas still need to be captured.

Another major direction is supporting spatial relationship queries, such as:

"the bag under the table"
"the chair beside the window"
"the object closest to the door"
"what is on top of the desk?"

This will require reasoning not only about what objects are present, but also about their positions and relationships in 3D space.

Eventually, we want users to ask broader questions about an environment rather than searching for only one object at a time. For example, Lumen3D could help users understand room layouts, locate misplaced belongings, inspect workspaces remotely, document indoor environments, or build searchable spatial memories.

Our longer-term vision is for Lumen3D to become a practical bridge between ordinary video, 3D perception, and natural-language interaction.