Spot-MIA

Inspiration

Boston Dynamics Spot robots walk inspection routes and report each mission as completed or failed. When we looked inside the data, 26 of 27 missions said completed, yet 21 of those had at least one failed inspection hidden inside. A status light that says "done" while a third of the checks failed doesn't earn an operator's trust. We wanted a tool that tells an operator what actually happened, where, and what to do next, without reading a log.

The name says it: things go MIA inside "completed" missions. Spot-MIA (Mission Intelligence Analysis) finds them.

What it does

Spot-MIA is an operator console for robot inspection missions.

Mission control: replay any mission on the facility map. The robot walks its route, each stop turns green or red, the battery drains toward a 30% danger line, and alerts stack up as failures and anomalies arrive.
Mission report: a headline ("7 of 10 actions succeeded") next to the robot's own COMPLETED stamp, a plain-English shift-log brief, one clear next step, and a ranked list of what needs a human.
Fleet trends: why missions fail, and whether performance is improving.

What we found in the data:

Completed doesn't mean it worked. 21 of 26 completed missions hid a failed action.
Battery is the biggest driver of failure. Below 30% battery only 49% of actions succeed, against 87% above it.
Low battery breaks sensors, not walking. 16 of 19 low-battery failures were sensor timeouts or unreliable readings. Navigation failures don't track battery at all.
V2's improvement is mostly battery. V2 raised success from 77% to 91%, but it also used half the battery per mission. At the same battery level, V1 and V2 perform about the same.
Problems hide behind successes. 8 anomalies sat on actions that reported success, including a critical abnormal temperature.

How we built it

Data pipeline: Python and pandas join missions, actions, readings, anomalies, locations and assets into one findings file. The dashboard only reads that file, and every number on screen has a note saying how it was computed.
Anomaly flags: each reading is compared against the normal range for its measurement type.
AI mission briefs: Qwen2.5-1.5B-Instruct, an open-weight model from Hugging Face, runs locally with transformers. Code builds a fact sheet for each mission, and the model only turns those facts into plain sentences.
Guardrails: about 20 checks in code reject any draft that adds a number or asset not in the facts, puts an asset in the wrong place, guesses a cause, adds an opinion, or flips above and below normal. Rejected drafts get one retry, then fall back to a template. 14 of 27 passed. The model never picks the next step and never grades its own work.
Front end: Streamlit and Plotly for the dashboard, plus an HTML and SVG reference console with an animated map replay.

Challenges we ran into

The small model made things up. It called a 52.9 dB reading "above normal" when normal is 72, and said the battery was "still sufficient". Instead of trusting it, we wrote checks for each kind of mistake and show the rejection count on screen.
Separating battery from configuration. V2 looked far better, but it also ran on fuller batteries and shorter missions. Comparing V1 and V2 at the same battery level showed the real story.
Incomplete event stream. The supplied replay file stops partway through mission M023, so we rebuilt the stream from the CSVs and checked it event by event against the original.
Running a model on a laptop CPU: version conflicts, a failed 3 GB download, and a nearly full disk, all solved in time for the briefs.

Accomplishments that we're proud of

Finding that V2's gain is mostly a battery effect, instead of reporting the headline number.
An AI feature that can't show an operator a made-up fact.
A console where an operator learns what failed, where, and what to check next in under 20 seconds.

What we learned

Small language models are good at wording, not facts. Code should decide what's true and check every sentence.
The first comparison is usually confounded. Always ask what else changed at the same time.
A mission status is a receipt, not an inspection.

What's next for Spot-MIA

Ask SPOT: operators type a question like "what keeps going wrong at Turbine Hall?" and get an answer drawn only from mission records, with the records shown.
Launch alerts: warn before a mission starts below 70% battery.
Live data: connect to Spot's API in place of the synthetic files.