Spotter

Inspiration

Most hands-on accidents and botched jobs don't come from not knowing what to do. They come from one skipped step. A caregiver mixes ORS with half the water. A food handler rushes hand-washing and skips the soap. A repair trainee opens a laptop that's still on charge. Checklists get ticked without anyone looking, and supervisors can't watch everyone. Chatbots can tell you the steps, but they can't see whether you did them. We wanted an AI that watches the work itself.

What it does

Spotter turns any camera (laptop, phone, or a recorded video) into a supervisor for any procedure:

1. Plan: type a task ("change a car tyre") or pick a preset. Tavily pulls authoritative procedures. Nemotron 3 Ultra compiles them into a checklist where every step has a visual success criterion (what the camera must see), a safety-critical flag and common mistakes. Sources are cited.
2. See: an on-device change detector only sends frames when the scene moves. Nemotron 3.5 Super VL judges each frame against the current step and the next ones, so it catches skipping ahead.
3. Act: Nemotron 3 Nano decides to advance, wait, warn, ask for a better angle, or raise an alarm, and speaks one short sentence (English or Hindi). Deterministic safety rules have the final say: a skipped safety-critical step always triggers a stop alarm (flash, beep, voice, plus an optional IoT webhook that can cut a smart plug).
4. Audit: each step keeps a timestamped evidence photo. Nemotron 3 Ultra writes a verdict, score, findings (including steps only marked done by hand) and coaching. Export as PDF or JSON.

How we built it

• Next.js on Vercel; all inference runs through Nebius Token Factory's OpenAI-compatible API, with one key for vision and three text tiers.
• The right Nemotron for each job: Ultra (550B MoE) for planning and auditing, where quality matters more than speed; Super VL for perception; Nano (30B-A3B, reasoning off) for the per-frame controller, where speed and cost matter. Each live-loop decision is more than 10× cheaper than routing everything through Ultra.
• "LLM proposes, rules guard": the controller's action passes through safety rules. A skipped critical step forces an alarm, safety-critical steps need ≥0.75 visual confidence, and low-confidence "advance" calls are held.
• Edge gate: 32×24 greyscale frame differencing in the browser plus a 6 s heartbeat. In our test runs it filtered 13–36% of frames before they reached the cloud, and more on a still scene.
• Live model trace in the UI shows which model did what and how long it took.
• Abuse and cost protection: per-IP and global daily caps, small token budgets, keys held server-side only.

Challenges we ran into

• Writing steps a camera can verify. "Make sure the water is clean" isn't observable. We made the planner produce a visualCheck for every step and forbid non-visual criteria.
• Real-time vs. reasoning. Nemotron 3 models reason by default. Turning thinking off for the perception and controller calls (chat_template_kwargs.enable_thinking=false) was key to a responsive loop.
• False advances. A single optimistic frame could tick a safety step. Confidence thresholds and rule-based guards fixed that without losing the model's flexibility.
• The model wanted to cry wolf. On first live runs, Nano raised a SAFETY STOP just because the hand-washing step wasn't visible yet. We changed the rules so only a rule-detected skip or a visible hazard can stop someone, and the model's 'hazards' that are really 'not visible' are filtered out. The model now coaches instead of alarming.

Accomplishments that we're proud of

• It works on any procedure with no task-specific training: from WHO hand-washing to RAM upgrades to ORS preparation.
• It catches a skipped safety step in real time and says exactly what was skipped.
• The audit is evidence-backed: a photo per step, with hand-overrides clearly marked as not verified.

What we learned

Model tiering matters more than model size. Ultra is excellent at turning messy web text into precise, observable criteria. Nano is fast enough to be the "reflexes". Vision models are much more reliable when you ask a narrow question ("is this step's visual check satisfied?") than an open one.

What's next for Spotter

• Run on a Jetson Orin Nano next to the work area as a fixed station, with on-device pre-filtering.
• Multi-camera views for occluded steps.
• Team dashboards for trainers: common failure steps across trainees.
• Offline packs for low-connectivity clinics.

Limitations (published on purpose)

About 1.5–3 s per decision, so it isn't for sub-second hazards. It needs a decent camera angle. It is advisory, not a certified safety interlock. Single view only, so occluded actions can be missed (manual override exists and is flagged in the audit). Rate limits are per serverless instance.

Built With

• nebius-token-factory
• next.js
• nvidia-nemotron-3-nano
• nvidia-nemotron-3-ultra
• nvidia-nemotron-3.5-super-vl
• react
• tavily
• typescript
• vercel
• web-speech-api