Operator In-Loop Novelty Labelling

Glorious front-end

Inspiration

Machine learning is a growing presence in intelligence, surveillance, and reconnaissance (ISR) particularly in applying neural network object detectors, localizers, and trackers to find threat objects such as visible/covert explosive devices or firearms. Even though I acknowledge the power of augmenting human judgment with AI, a major problem is that machine learning models struggle to adapt as the problem space changes. For example, a shift in the appearance and type of materials used to make improvised explosive devices could be catastrophic to a static model. Often you will want the organization deploying the AI to send the data back to the research team to update to ML. However, if the model has been running for a month, the research team has to watch a whole month of video footage.

The big question: what can you do to better your chances at getting the best improvement to your AI as fast as possible?

The big solution: have operators collect only the most valuable data while they are fielding it

What it does

It is a mimic display that deploys MobileNet, a preconfigured neural network object detector. My display, however, augments the typical view with addition features that allow the operator to choose what data to save to disk.

a button to start recording video to disk and declare a new threat object
an input box for annotating a novel threat object
a bar graph for prioritizing the most frequently seen (and therefore more pressing) novel threat objects

How we built it

We set up a PiCam/RaspberryPi pair to work as a telemetry station and stream video over a socket. Our laptop receives the data and connects it to a dashboard built in Plotly/Dash. On the backend, the dashboard runs object detection on the video feed and updates the display if an object is found (Keras/Tensorflow).

Challenges we ran into

Sockets. Getting 2 separate devices to communicate with each other fluidly and reliably. Getting everything to run real-time. How much do we have to slow the ML down to maintain a real-time video? Is this rate of ML good enough?