Inspiration
LiDAR captures streets and infrastructure as millions of colored 3D points. Those points contain useful information, but turning them into recognizable objects takes work. We wanted to explore how AI and 3D shape measurements could help people inspect this data more efficiently.
What it does
Our prototype helps users find and review possible objects within a large point cloud.
It separates ground points, groups nearby points into object candidates, and retrieves more original points to make those candidates easier to see. Users can inspect different views and record an object label, crop quality, and notes.
We also explored two sources of evidence: CLIP’s suggestions from rendered images and measurements of each candidate’s 3D shape.
How we built it
We built the workflow with Python and Jupyter notebooks, using a smaller sample to explore the scene before retrieving detailed points around selected candidates.
We rendered candidates from four angles and used pretrained CLIP to compare those images with possible object descriptions. Separately, we measured whether local point patterns were straight, flat, or scattered.
Finally, we built a browser-based review tool so people could inspect candidates and save their own assessments. Model suggestions stay hidden until requested, allowing reviewers to make an initial judgment independently.
Challenges we ran into
Finding a group of points did not always mean finding one complete object. Some candidates were fragments, while others included surrounding ground, wires, vegetation, or attached structures.
CLIP’s suggestions were also inconsistent. A pole-like candidate could rank “bicycle” above “utility pole.” This showed us that a model’s output depends heavily on how the data is presented.
Large files and notebook visualization issues added practical challenges. We used cached data and standalone browser views to make exploration easier.
Accomplishments that we're proud of
We built a working path from a large point cloud to a human review interface.
Retrieving more original points revealed structures that were difficult to recognize in the smaller sample. We also created a review tool with 20 candidates, saved assessments, and review history.
We kept model suggestions separate from human labels and documented uncertainty instead of treating every prediction as a confirmed result.
What we learned
More detailed data made candidates easier to inspect, but better object boundaries still mattered.
We learned that matching the model to the data is essential. CLIP provided an exploratory baseline, but its suggestions were not reliable enough to use without review. Shape measurements added useful evidence, although they did not establish an object’s identity on their own.
What's next
Our next step is to collect reviewed labels and identify which candidates contain usable, sufficiently complete objects.
With enough reviewed examples, we plan to train a model using 3D shape measurements. We will compare image-based predictions, shape-based predictions, and a combination of both on separate spatial groups.
Training and that evaluation are future work. After testing the smaller workflow, we aim to expand to a larger area.
Log in or sign up for Devpost to join the conversation.