Inspiration
Video can be more than just the backup plan.
We think that when you finally decide to try taking something apart, you shouldn't have to worry about how to put it back together. That's what Rebuilt's goal is. To let you turn a quick handheld video into steps you can refer to later.
What it does
Rebuilt tracks parts throughout a build's timeline to detect changes, it then uses those detected keypoints to generate a final set of instructions.
How we built it
We started by identifying the core challenge for our system: tracking piece connections. Everything following that was different explorations of that problem to see how we could balance for quality, speed, and reliability.
The problem is divided into two stages: detecting connections, and guide generation. Our approach for the first stage uses the final frame of disasembly alongside meta's SAM 2 model to track the pieces backwards. We use the motion of these boxes to create a list of candidate timestamps for connections.
This candidate list is then sent to a VLM for post-processing. This is where we try and root out false positives from our detection algorightm, such as corners overlapping, but no actual shared area between parts. From here, we have a list of changes, with timestamps in the video that we can then use as a final instruction guide.
Challenges we ran into
We found that we really had to optimize the segmentation models we were running for this, and that they took a lot more tuning to get the results we desired than originally anticipated.
We tried a lot of different hybrid systems for detection, including combinations of VLMs, segmentation models, SLAM, etc..
Accomplishments that we're proud of
This project has shown us that
What we learned
Ideas are easy, optimization is hard. This is a deep rabbithole, but a promising one.
What's next for Rebuild
We were surprised when our weekend hack turned into such a tricky problem. Our solution has lots of room for improvement in it's algorithms, and we can see tools like this becoming very useful for real hardware projects if a reliable enough implementation is reached.
Built With
- openai
- sam3
