Inspiration
During robotics meetings, we are doing one of three things: solving problems (best case), looking for something, or wondering why what we did broke the robot. The main issue is communication. When one team member changes something that they think is minor, it's no big deal. When several team members incrementally change “minor” items, they compound into bigger issues that turn into catastrophes. This problem is one that many engineering teams face, whether it be in data centers, factories, or oil refineries. We wanted to produce a system that uses measured data from multiple members of the same team to build a ground truth for what has happened, when it has happened, and the implications for others.
What it does
Every member of an engineering or maintenance team wears something analogous to a body cam. These cameras autonomously identify and document actions, including the use of key tools and equipment. Each individual camera streams video to both an object identification model and a vision-language model. The object ID model is responsible for recognizing and documenting the placement and updated locations of important tools. The vision-language model builds a timeline of events for each worker, which, when put together on a centralized server, makes a well-documented paper trail for every action, decision, and possible mistake made. This centralized server is queryable directly through text, and rules can be written by users to autonomously flag and respond to certain events. For example, completing a certain HVAC maintenance task may automatically take items off a to-do list or alert workers to start new tasks.
How we built it
We built ReadyCam around a central FastAPI server that stores timestamped observations, camera identities, and supporting frames in SQLite. A Raspberry Pi agent samples short windows of video and sends them to the server. NVIDIA Nemotron turns those frames into draft descriptions, while a separate tool identification workflow lets us label images and train a YOLO model for objects that matter to a particular team. The dashboard brings individual workers’ observations into one timeline and lets users ask questions against the recorded evidence. We also added support for Limelight AprilTag data so observations can eventually be tied to positions on a map.
Challenges we ran into
The hardest part of building this was creating a reliable record from footage that is often incomplete. A camera may see someone reach toward a tool without showing whether they picked it up, or capture an adjustment without proving that a repair was finished. We had to keep the original frame attached to each observation and treat model descriptions as drafts. Getting the hardware connected was another challenge: the Pi can communicate with the server, but we are still working through camera detection and the Limelight’s video connection.
Accomplishments that we're proud of
We built the shared memory system, a dashboard for camera and worker timelines, and text queries that return supporting observations. We also got the Raspberry Pi agent running as a service and communicating with the Mac server over a direct Ethernet connection. Nemotron successfully described sample visual data, and we built the image labeling and training workflow needed to teach the object detector about a team’s own tools. The parts are designed so a new camera or work area can be added without rebuilding the whole system.
What we learned
We learned that remembering an engineering task is different from summarizing a video. A useful record has to preserve who saw something, when they saw it, and the image that supports the claim. It also has to distinguish an observed action from an assumed outcome. We learned that customization matters just as much as the model: the important tools, locations, and task steps will be different for a robotics team, a factory, and a maintenance crew.
What's next for ReadyCam
Our first priority is getting a continuous live camera feed into the full pipeline and demonstrating several workers contributing to the same timeline. After that, we want to train the detector on our own tools, improve location tracking with AprilTags, and add rules that flag missing steps or important changes between shifts. Longer term, we want ReadyCam to turn these observations into a draft handoff that workers can check and correct before the next team takes over.
Built With
- ai
- computervision
- limelight
- nemotron
- python
- raspberrypi
