Kairos

Catch the puddle. Clear the storm. A weather and water plugin for Orbit: Spot gets RECALL when a storm turns mid-round and HOLD when its camera sees water in the path.

Comment

Catch the puddle. Clear the storm. A weather and water dashboard for Boston Dynamics Spot, built as a plugin for Orbit, for the NextEra Energy "SPOT Management Dashboard" challenge.

Live: https://kairos.boston

Inspiration

NextEra runs Boston Dynamics Spot robots on inspection rounds at its plants. I talked with NextEra's Spot team at the Knight Hacks career fair and at their workshop, and they named two problems. Weather on the route is not caught as early as they want, and a storm that builds mid-round is the hard case. Small puddles and spills on the route are also hard to catch.

Spot is rated IP54: light rain and splashes are fine, harsh rain and storms are not. Orbit, Spot's fleet software, can block a launch on a rain forecast. The gap is after launch. Kairos keeps checking rain, thunder, gusts and heat against Spot's limits while the round runs, and calls a recall when they cross. No forecast can see a puddle in Spot's path, so Kairos also watches the floor.

What it does

Kairos watches the weather and the floor on every round. When the weather turns mid-round, it says RECALL. When Spot's camera sees water in its path, it says HOLD until an operator confirms. Every call is logged.

  • Dashboard: the call for a Spot on route right now (RECALL / HOLD / GO), the rain rate against Spot's IP54 limit, thunder, gusts and heat, a 12-hour rain chart with the next safe launch, live radar around the plant, the latest water Spot's camera found (Confirm / Reject), and the next round's call.
  • Live round: one simulated round. Spot walks its route, its camera checks every waypoint, a spill in the path holds Spot until the operator answers, and the recorded storm recalls Spot to the dock.
  • Storm drill: a real pop-up storm at St. Lucie (May 19, 2024), replayed minute by minute with no hindsight. The recall fires before the NWS warning (first on my thunder rule, which is labeled as my assumption).
  • Water model: try my model on your own photo, in the browser, with honest held-out numbers and what each confidence % means.
  • Evidence and Incident review: every water detection and operator label, stored in Neon; each model version scored on a frozen Neon branch; saved moments read back from Neon snapshot branches.

A scenario switch in the top bar replays a real St. Lucie storm I captured Sat Oct 10, 5:56 PM ET (NWS + Open-Meteo + 50 minutes of radar), so the demo works whatever the sky does. Live uses the feeds right now. The demo site is NextEra's St. Lucie plant; Kairos works for any plant from its coordinates.

How it fits Orbit: Kairos is a layer on top of Orbit, not a replacement. A HOLD maps to pausing the schedule and a RECALL to sending Spot to the dock (Orbit API v0 endpoint names, simulated in this demo). The water model runs wherever Spot's camera frames are: in the browser today, on Spot's CORE I/O as a Network Compute Bridge worker in a real deployment.

Rules (all in lib/verdict.ts, cited in the UI)

  • Rain above 2.5 mm/h before Spot can dock (beyond IP54's "light rain") → HOLD before launch, RECALL during a round.
  • NWS warning → HOLD / RECALL. NWS watch or advisory → HOLD.
  • Thunder chance ≥ 30%, gusts ≥ 50 km/h → HOLD / RECALL (my assumptions: the Spot guide is silent).
  • Air temperature within 5 °C of Spot's -20 / 55 °C limits → HOLD / RECALL.
  • Water in Spot's path → HOLD at that waypoint until an operator confirms or rejects. An unknown floor (model loading or failed) also holds.
  • Missing or stale weather never shows GO.

How I built it

  • Next.js 16 on Vercel, TypeScript, Tailwind, shadcn/ui. Rules are pure, tested functions (lib/verdict.ts, npm test), so the live dashboard, the storm drill and the backtest run the same code.
  • Hugging Face: my water detector is fine-tuned from RT-DETRv2-R18 (Apache 2.0) on Hugging Face Jobs (one L4 GPU, 18 epochs, about $1 of credits), published on the Hub with an honest model card, and run in the browser with transformers.js (WebGPU fp32, WASM int8 fallback). Without it, Kairos can see the sky but not the floor.
  • Neon: Neon Postgres is the evidence log (every detection and label), a frozen branch scores each model version, every visitor and every storm drill writes to its own copy-on-write branch so nobody's clicks touch production, and snapshot branches power Incident review. Without it there is no audit trail, no model scoring, and a public demo anyone could break.
  • NextEra (the challenge): the problem came from their Spot engineer, and the robot data is NextEra's synthetic SPOT challenge kit (labeled synthetic in the UI).
  • Weather: National Weather Service (api.weather.gov) alerts and forecast grid, Open-Meteo 15-minute rain, NEXRAD radar tiles via the Iowa Environmental Mesonet, map tiles © OpenStreetMap contributors.
  • Data for the model: split by source and scene, never by single image. Test photos come from a puddle source the model never trained on, plus whole recordings from a camera on a real Spot (UT Austin SCAND). The threshold was picked on a separate calibration split, never on test.
  • Domain: kairos.boston, a GoDaddy Registry TLD.

The numbers (all on public photos; no NextEra camera frames exist)

  • Water model, held out (never trained on), fp32 (the WebGPU file): flags 85% of wet photos from a puddle source it never saw (463 of 543); 1.2 false alarms per 100 frames from a camera on a real Spot (2 of 172). Known gap: tiny puddles under 1% of the frame, 6 of 57.
  • Confidence that means something: "water 90%" means 9 in 10 boxes at that score were real water, measured on a calibration split (315 boxes) and checked on test (680 boxes).
  • Plant-like floors (int8 file): on the industrial-like test slice (indoor hard floors and spills), it flags 86% of wet frames (245 of 284) with 2.4 false alarms per 100 indoor Spot frames (4 of 165).
  • Speed: about 70 to 100 ms per frame on WebGPU in Chrome (two measurements: 67 ms p50, and 101 ms median of 10 runs).
  • Weather backtest: at the kit's 27 round times, on archive weather for the plant (Open-Meteo ERA5, Sep 1-10), Kairos would hold 1 round for weather. It does not cry wolf. The archive has no thunder data, so this does not test the thunder rule.

Full tables, splits and 95% intervals: docs/model/water-v2/RESULTS.md and CONFIDENCE.md in the repo.

Challenges I ran into

  • No weather and no real camera frames in the kit. NextEra's kit is synthetic and indoor, with no weather fields. Every weather assumption (outdoor leg, dock, walking speed, my thresholds) is labeled on screen, and every camera frame is a public photo with its license.
  • A score that was too good. My first water model (a ConvNeXt classifier) scored 99.6%, because wet and dry images came from different datasets. The honest single-source number was 92.7%. That is why v2 splits by source and scene and picks its threshold on a separate calibration split.
  • Tiny puddles are hard. Under 1% of the frame, v2 finds 6 of 57. I tried a second fine-tune (v3) with bigger inputs, copy-paste of tiny water and mosaic; it lost to v2 on every recall metric, so v2 stays and I say so.
  • A rain-at-the-door camera model that learned "gray sky". After deduplicating its splits it was near chance on cameras it never saw, so it gates nothing. Weather data decides.
  • Browser inference. The transformers.js object-detection pipeline drops every box of a one-class model, so I decode the raw outputs myself. The int8 file ran at 1.3 s per frame on WebGPU (no int8 matrix kernel); fp32 on WebGPU runs at about 70 to 100 ms. I also patched two float64 casts in the exported graph that browsers cannot run (outputs identical).

Accomplishments that I'm proud of

  • NextEra's Spot engineer gave me the two problems at the workshop, liked the idea, and asked to hear back as I build it.
  • A replay of a real storm at the real plant where Kairos recalls Spot before the NWS warning, with no hindsight.
  • An honest model card: the number that is not flattering (tiny puddles) is in the headline, and a confidence % that is measured, not a raw score.
  • A public demo that cannot be broken: every visitor writes to their own Neon branch.

What I learned

  • How Spot actually runs: Autowalk missions can keep going when the link drops, so a recall must be sent while Spot is still in coverage; the radio range drops behind concrete; operators return Spot to charge around 20% battery.
  • A clean benchmark score can come from the dataset, not the model. Split by source and scene.
  • Shipping a model in the browser means owning the decode, the precision and the export, not just the weights.

What's next for Kairos

I want to take Kairos past Knight Hacks. NextEra's Spot engineer asked to hear back as I build it, so the next steps are the ones that need a real plant:

  • Train and test on NextEra's real Spot footage and the operator labels Kairos already stores in Neon, to close the tiny-puddle gap on plant floors.
  • Run the water model on Spot's CORE I/O as a Network Compute Bridge worker, next to NextEra's own models.
  • Wire HOLD and RECALL to the real Orbit API instead of the simulated calls, and pilot it on one site's outdoor rounds.
  • Add a dock-door camera check, so Kairos sees rain at the door, not only in the forecast.

Disclosure (Knight Hacks rules)

All project code, both fine-tuned models and the data pipelines were made during the event (hacking began Fri 2026-10-09, 11 PM EDT). The repo's first commit (Fri 7:54 PM) is GitHub's empty repo setup only (a title line and .gitattributes).

  • AI tools: Claude Code (Anthropic Claude models) wrote most of the code under my direction; OpenAI Codex reviewed facts and evaluations. I chose the problem, the rules and every product decision, and reviewed and tested the output. The intro narration is an ElevenLabs voice ("Misa").
  • Base models: PekingU/rtdetr_v2_r18vd (Apache 2.0), fine-tuned by me into builtbyd3v/kairos-water-rtdetr. Earlier experiments: facebook/convnext-tiny-224 (Apache 2.0) into builtbyd3v/spot-wetfloor-convnext-tiny and builtbyd3v/kairos-door-rain.
  • Datasets: Bhutad and Patil (2021), Dataset of Stagnant Water and Wet Surface with Annotations, Mendeley Data V4, doi:10.17632/y6zyrnxbfm.4 (CC BY 4.0); Roboflow Universe slipp/spill_detection, hk-hsse/puddle-qrpyh and projects-pwwbs/water-puddle-joyj6 (CC BY 4.0); Kaggle muftirestumahesa obstacles in public spaces (CC0); UT Austin SCAND, Karnan et al., doi:10.18738/T8/0PRYRH (CC0); for the door-rain experiment, Kaggle Weather Image Recognition (CC0), Multi-class Weather Dataset (CC BY 4.0) and Weather-Time (CC0).
  • Challenge data: NextEra Energy's SPOT KnightHacks Challenge Kit (synthetic and fictional).
  • APIs and data: National Weather Service api.weather.gov; Open-Meteo (CC BY 4.0); NEXRAD radar tiles via the Iowa Environmental Mesonet; map tiles © OpenStreetMap contributors.
  • Libraries and services: Next.js, React, Tailwind CSS, shadcn/ui, Radix UI, lucide-react, Geist, Hugging Face transformers.js and Jobs, Neon Postgres, Vercel. Domain via the MLH GoDaddy Registry offer.
  • Kairos is a student hackathon project. It is not affiliated with or endorsed by NextEra Energy or Boston Dynamics. Spot and Orbit are trademarks of Boston Dynamics.

Built With

Share this project:

Updates

Submission history