FieldLens

Inspiration

The skilled trades are short over 100,000 technicians, and the experts are retiring faster than replacements arrive. New techs get a week of lectures, then work alone, where the expensive mistakes are the ones nobody knew to call about. What if every rookie had a senior tech watching their first hundred jobs? And what if one real supervisor could see every rookie at once?

What it does

OVERWATCH is an agentic spatial supervision system with two surfaces over one event stream:

Rookie View is the trainee's camera view. The Guide agent streams their job checklist. The Watcher agent analyzes the video live, drawing instruction and hazard boxes on the footage and advancing steps as work completes. The Coordinator speaks step-by-step coaching and interrupts on safety risks.

Command Room is a spatial multi-view control room on PICO: floating panels for each site feed, the Triage agent's live attention ranking, the Guide's auto-drafted corrective docs, and the Scribe's audit log. When a Watcher catches a hazard, the room reacts in a visible chain: box lands, narration fires, priorities re-rank, and the fix drafts itself. The supervisor directs the room by voice.

Five agents, in parallel, visible and in motion, each doing something different at once.

How we built it

React + TypeScript + Vite, with the WebSpatial SDK for spatial multi-window rendering, running in the PICO Emulator as shell-floated windows synced through a BroadcastChannel event bridge. All agent intelligence is live Gemini vision and function-calling. Frame analysis, checklist generation, triage ranking, and voice command routing are real model calls (technician footage is representative sourced clips). A deterministic demo mode with cached analyses makes the choreography reproducible, and 79 unit tests cover the event bus, cascade timing, step-advance logic, and validators.

Challenges we ran into

The emulator's browser doesn't expose the WebSpatial runtime globals. We discovered this with devtools probing over adb, then built the multi-window experience with shell-floated browser windows and our sync bridge instead of scene APIs. Free-tier model quotas (20 requests per day) forced a cache-first architecture with surgical live-call budgeting. And LLM SSE framing, vision box coordinates, and browser tab throttling each cost us a real debugging session.

What we learned

Spatial multi-view isn't decoration for agentic systems. It's what makes orchestration legible. Watching five agents react to each other across floating windows communicates "agentic system" faster than any architecture diagram.

What's next

Deploying Guide's instructions to AR glasses on the job site, real multi-site video ingest, and world-model spatial understanding of the workspace so agents reason about equipment, not just frames.

Built With

• pico
• react
• typescript
• webspatial