ElideDb

Umap Representation of the HDBScan
Semantic Query Retrieval System

Inspiration Robotics and Frontier Research AI developers are drowning in large amounts of unannotated data. Every single hardware test rollout or simulation run generates massive, unindexed multi-modal streams: multi-angle RGB video, LiDAR point clouds, and high-frequency sensor telemetry. Currently, if an analyst needs to find a specific edge-case failure, they have to manually watch hours of test footage or spend 3-6 month on annotating the data whose requirements are going to change midway through the development. We wanted to build the infrastructure layer that eliminates this friction, turning raw pixels and physics into an instantly queryable asset.

What it does ElideDb is a specialized multimodal semantic vector database built specifically for physical AI. Users simply drop in raw, un-annotated video files and sensor logs with timestamps. ElideDb automatically segments the videos and aligns the visual patches with the corresponding time-series sensor telemetry, mapping them into a shared 1056-dimensional embedding space. The system then clusters these embeddings to create an organized, hyper-efficient index. Engineers can then use natural language to search their entire dataset for highly specific physical interactions like "pedestrian interacting with vehicle" retrieving the exact clips across thousands of hours of data instantly.

How we built it ElideDb has two layers: a C++20 storage engine for speed, and a Python/MLX pipeline for the ML.

The engine uses two custom file formats. SDX stores sensor telemetry in sorted, chunked columns with min/max stats in the footer, so time queries skip everything irrelevant. SFI indexes each video's keyframe structure, mapping every GOP to its exact byte range. A query for a 2-second window reads a few hundred KB instead of gigabytes. Video is never re-encoded; frames come back as tensors.

The semantic layer runs fully on-device. SigLIP (via MLX) embeds video windows, HDBSCAN clusters them into scene types with zero annotation. A text query matches clusters first, then ranks by cosine similarity, and the results compile down to tiny byte-range reads. Streams stay raw at ingest; alignment happens at query time.

Challenges we ran into Compressed video fights random access: you can only decode from a keyframe, so we indexed at GOP granularity and verified byte-offset seeks produce correct frames. Sensor clocks drift, so we store per-stream offsets as metadata instead of rewriting timestamps. And our first retrieval design used UMAP distances, which turn out to be meaningless. We demoted UMAP to visualization and moved ranking to the original embedding space.

Accomplishments that we're proud of Queries read under 1% of corpus bytes, and the counter in the demo is real, instrumented at every I/O path. Correct clips returned for queries like "pedestrian interacting with vehicle" over footage nobody ever annotated. Two working file formats we can show in a hex dump. And the whole thing runs on one MacBook, no cloud, which matters for teams whose data can't leave the building.

What we learned The bottleneck in physical AI isn't models, it's that databases assume data is small and comparable. Video breaks that, so teams store URLs and the database goes blind: no stats, no pruning, every read is a full read. The fix is making the inside of multimodal data visible to the query layer, and being honest about it: if every byte isn't counted, the elision number is fiction.

What's next for ElideDb We plan to scale ElideDb to natively support 3D LiDAR point-cloud embeddings alongside video patches. We are also aiming to build native integrations directly into popular simulation frameworks like NVIDIA Isaac Sim and ROS2 (Robot Operating System), allowing robotics teams to pipe their test logs directly into ElideDb in real time as they run experiments.