Scout

Inspiration

We asked: what if we turned the project's drawings into a wearable map that doesn't just guide — but watches for hazards?

What it does

Scout is a wearable navigation and real-time hazard-awareness system for Meta smart glasses. It converts a site's own drawings into a queryable graph in the site's local coordinate system, then uses the glasses' camera, mic, and motion sensors to figure out where the wearer is and guide them by voice — while flagging drop-offs, moving equipment, and closed zones the instant they come into view.

How we built it

We grounded the architecture in published research — frontier-based navigation (OpenFrontier, ETH Zürich 2026) adapted for humans instead of robots, combined with classical computer vision (optical flow, monocular depth segmentation) for hazard detection running locally on the paired phone. The positioning system deliberately avoids GPS-as-lookup; instead it uses dead reckoning anchored to the site's own surveyed local grid, the same coordinate system construction already uses. Drawing ingestion is OCR + object detection + VLM fusion to avoid hallucination. Everything runs on a zero-budget stack (Gemini Flash, Groq free tier, local models via Ollama).

Challenges we ran into

The trickiest part was reconciling cloud-dependent VLM calls (for frontier scoring, signage reading) with real-time hazard detection on a head-worn camera under poor site connectivity and frequent head motion. We solved it by layering: negative-obstacle and moving-equipment detection run locally at video rate on the phone (independent of connectivity); cloud VLM calls come only at navigation decision points. Optical flow needed ego-motion compensation using IMU data so it actually detects moving equipment instead of flagging the whole world. Battery life under continuous audio + periodic video is tighter than the headline 8-hour spec — we had to design active duty-cycling (camera dormant outdoors where GPS works, active only indoors where frontier guidance matters).

Accomplishments we're proud of

The relative-positioning philosophy — using the site's own local survey grid instead of inventing a new one, anchoring it with a single GPS calibration point, then tracking displacement rather than absolute position. It's simple, it works where GPS fails (underground, under slabs), and it reuses infrastructure construction already has. Also: the honest limitations section. We named every seam (barometric floor detection in open structures, dead reckoning drift on stairs, unconfirmed PM-platform webhook access) rather than glossing over them — judges respect that more than overclaiming.

What we learned

Training-free perception (general VLMs + classical CV) beats domain-specific models when no construction-site training data exists yet — but you have to separate concerns carefully. Don't let one cloud call do two jobs. Dead reckoning on a moving camera needs ego-motion compensation or it fails silently. And the Stage 1 scope discipline from the procurement-track sibling (build one thin vertical slice end-to-end, don't optimize pieces in isolation) applies just as hard to hardware+software stacks — integration risk is real.

What's next for Scout

Immediately: deploy on an actual construction site (past the staged demo), validate frontier detection and hazard flagging on real messy drawings and live equipment. Stage 2 adds the restriction layer (supervisor-confirmed exclusion zones), crowd-sourced path refinement (multiple workers' traces converge on ground truth), D* Lite replanning for mega-project scale, and multilingual voice. The bigger win: this architecture doesn't depend on Meta hardware — it ports to ruggedized industrial AR glasses (RealWear, Rokid) for real ANSI Z87.1-certified deployment. And once multiple projects use it, the repeated-game framing (same vendor pool renegotiated across phases) that the procurement track explores for vendors applies here too — workers on the same GC's portfolio of sites start teaching each other's routes.