ForeSight: AI-Augmented Wearable Intelligence for Autonomous Site Supervision

WnCC × Kaya AI · IIT India Hackathon 2026 · Track: Physical AI (Wearables / Computer Vision / Site Intelligence) Team Snowman: Aishee Lahiri (IIT KGP), Mridini Mittal (IIT KGP), Suhaan Aneja (IIT KGP)

Inspiration

The idea for ForeSight didn't start with a model or a piece of hardware; it started with a document. Every construction and industrial site we looked into runs on a Standard Operating Procedure that someone wrote once, carefully, in a PDF or a binder, and that almost nobody re-reads once they're on a scaffold with gloves on. The procedure that actually gets followed is the one a worker remembers, not the one that was written down. That gap is invisible right up until an inspection or a failure surfaces it, and by that point the cost of fixing it has already multiplied.

We kept coming back to a simple, slightly uncomfortable question: on a modern construction site, everything upstream of the work (design, estimation, procurement) is digitized, and everything at the point where a plan actually meets a worker's hands is still verified by memory and a walkthrough. India's ongoing infrastructure build-out only makes that gap more consequential: more sites, more crews, more SOPs nobody has time to re-check line by line.

What made this feel buildable, and not just a nice problem statement, was the recent maturity of two things at once: vision-language models that can be asked a grounded yes/no question about a single frame with real reliability, and wearable hardware (smart glasses with a camera, a mic, and a compass) that no longer requires bespoke industrial hardware to prototype against. That convergence is what made us want to build this specific system, rather than another dashboard that shows you a problem after it already happened.

What It Does

ForeSight puts an AI foreman on a worker's shoulder. A worker checks in by voice at the start of a task, and the system:

Speaks the current step, sourced from the project's own SOPs, parsed once at project setup rather than hard-coded.
Identifies materials by reading their printed marking, never by guessing from appearance, and reads out the relevant spec or usage note in the same breath.
Verifies step completion visually, comparing a sampled frame against what a correctly completed step should look like, before allowing the worker to move on.
Logs every material used automatically, feeding a live stock count that drafts a reorder before a shortage becomes a stoppage.
Listens and watches for deviations from a worker's own established rhythm: a personalized motion-and-sound baseline, not a generic one.
Watches continuously for a defined set of hazards, and warns a worker directly, immediately, if their own heading points toward one they can't currently see.
Escalates anything unresolved to a supervisor through a single channel, triaged by a fixed, explainable urgency table, never a model's best guess about how urgent something is.

In short: it turns a static SOP into a live, spoken, continuously-verified supervisor that never gets tired of checking.

How We Built It

System Architecture

We organized the system into three layers, and we were deliberate about keeping the boundary between them clean, because that boundary is what makes the system debuggable under pressure: both ours, during the build, and a judge's, during questioning.

Layer 1: Edge Perception & Document Ingestion. Smart glasses stream frames, audio, and heading data. Separately, at project setup, an LLM-based document-intelligence pipeline parses the project's existing SOPs into a structured, step-indexed task pack, a one-time ingestion step per project, not per worker. If connectivity drops mid-verification, the system falls back to on-device frame-differencing against a captured before/after reference pair, a non-AI, purely local check that keeps the guidance loop alive with zero network dependency.

Layer 2: Core AI Orchestration. This is where the actual decisions happen, split across five cooperating segments:

Task & Step Guidance: voice-driven step walkthrough with VLM-based completion checks.
Multimodal Anomaly Watch: per-crew motion + sound baseline, scored independently per channel.
Resource Management: consumption logging against a manually-seeded stock count, with a simple depletion forecast.
Escalation: the traffic controller, packaging context and routing every handoff by a fixed urgency lookup.
Passive Hazard Watch: continuous scanning for a bounded hazard list, using each worker's own glasses heading for proximity instead of installed beacon infrastructure.

Layer 3: Notification & Ledger. A single fixed supervisor channel (WhatsApp), drafted for one-tap approval on routine items and sent immediately, without a review step, for hazard-tier events, plus a structured ledger that logs every handoff and its eventual resolution.

Working Principle

The design choice we're proudest of is where we drew the line between judgment and routing. Some decisions genuinely need a model's interpretation of unstructured input: does this frame match a completed step, does this marking identify this material, does this pattern match a known hazard. Other decisions, specifically how urgently something should interrupt a human, we deliberately kept out of the model's hands entirely and encoded as a fixed, auditable lookup table instead. If a hazard is detected, its severity tier is pre-assigned by hazard type; if a step goes unresolved, its urgency is pre-assigned by handoff type. A safety-relevant routing decision, in our view, has no business being a probabilistic guess a model might get wrong on a bad frame.

The step-verification chain is a good example of how we built in graceful degradation rather than a single point of failure:

$$ \text{Step verified} = \begin{cases} \text{Pass, log material & advance} & \text{if } P_{\text{VLM}}(\text{step complete}) \geq 0.70 \ \text{Compare against reference-completed frame} & \text{if } P_{\text{VLM}}(\text{step complete}) < 0.70 \ \text{Mark unresolved, escalate to Segment 4} & \text{if reference comparison is also inconclusive} \end{cases} $$

with an independent, connectivity-triggered fallback to on-device frame-differencing whenever no network call can be made at all: a separate branch, not a fourth tier of the same confidence chain.

Resource forecasting deliberately stayed simple, on purpose. We projected depletion using a moving average over recent logged usage rather than any kind of learned forecasting model:

$$ \bar{c}n = \frac{1}{n}\sum{i=1}^{n} c_i, \qquad \hat{t}_{\text{depletion}} = \frac{S_t}{\bar{c}_n} $$

where $S_t$ is current stock and $c_i$ is material consumed in the $i$-th most recent logged interval. A reorder is drafted, never auto-sent, once $\hat{t}_{\text{depletion}}$ falls inside a manually configured lead time. "A moving average" is a less exciting answer than "AI predicts it," but it's one we could actually defend line by line if asked how it works.

The anomaly watch treats its two channels asymmetrically, because they aren't equally trustworthy in a noisy site environment:

$$ \text{Flag}{\text{high-confidence}} = \mathbb{1}!\left[d{\text{motion}} \geq \tau_m\right] \wedge \mathbb{1}!\left[d_{\text{sound}} \geq \tau_s\right], \qquad \tau_m < \tau_s $$

Motion is the primary signal and needs to cross a lower deviation threshold $\tau_m$ to count; sound is corroborating and needs a higher threshold $\tau_s$, since it's far more prone to ambient site noise. A single channel deviating alone stays low-confidence and is logged, not discarded, but if the same channel deviates across several consecutive reps, we escalate that streak on its own, since a persistent single-channel pattern is meaningful even when one instance isn't.

Tech Stack

Wearable capture: smart-glasses form factor, onboard camera, microphone, and compass/heading sensor as the three input streams.
Vision-language understanding: an off-the-shelf multimodal LLM/VLM, used only for the two grounded tasks it's reliable at: frame-level step/hazard verification and printed-marking-based material identification, never for open-ended visual judgment.
Document intelligence: an LLM-based parsing pipeline that structures a project's existing SOPs and spec sheets into a step-indexed task pack at ingestion time, document parsing and structuring, not model training.
Speech pipeline: automatic speech recognition for the voice check-in and clarification loop, paired with text-to-speech for spoken step instructions and hazard warnings.
Offline fallback: on-device frame-differencing against a captured before/after reference pair, classical CV, no model call, the path that keeps the core loop alive without connectivity.
Backend & rules engine: a lightweight service layer implementing the fixed-lookup urgency and severity tables that drive Segment 4 and Segment 5, deliberately deterministic, not model-driven.
Resource tracking: a simple stateful tally per material, decremented by logged consumption, with moving-average depletion projection as derived above.
Notification layer: WhatsApp Business API integration for one-tap supervisor approval and immediate hazard-tier alerts.
Ledger: a structured, append-only log of every handoff, detection, and manually-recorded resolution, the foundation a Phase 2 dashboard would read from.

Challenges We Ran Into

Resisting the urge to make everything a model call. Our first instinct for "classify urgency" was to reach for an LLM. We talked ourselves out of it deliberately: a safety-relevant routing decision needs to be explainable if it's ever questioned after the fact, and "the model decided" is not an answer we were willing to give. Replacing it with a fixed lookup table was a harder design discipline than it sounds, because it meant admitting a model shouldn't touch a decision we could otherwise have bolted an LLM onto in an afternoon.
Killing our own beacon-triangulation plan. Our original design for hazard-proximity detection assumed installed beacon hardware across the site. It was the cleanest version of the idea and the least buildable one: it assumes infrastructure a hackathon (or most real sites) simply doesn't hand you. Reworking hazard proximity around only what's already on the glasses (last checked-in location and live compass heading) took real back-and-forth, because it meant shipping a smaller, honest claim instead of a bigger, unbuildable one.
Confidence thresholds don't fail gracefully by default: you have to design that in. Our first version of Segment 1's verification chain just had a single VLM check with no fallback, which meant any low-confidence frame turned into a dead end. Building the reference-image comparison and offline frame-differencing tiers underneath it, and making sure they triggered on genuinely different conditions (confidence vs. connectivity, not the same one), took more iteration than the "happy path" demo suggests.
Segment 2 is real, but it's not demo-friendly. The per-crew motion+sound baseline needs a live calibration pass before it can flag anything, which is a bad fit for a cold, one-off hackathon demo. We had to be honest with ourselves that this piece would need to be shown as a scripted walkthrough rather than performed live, and we said so plainly instead of quietly hoping no one would ask.
Knowing what to leave out. The hardest challenge wasn't technical at all, it was scoping. Predictive procurement, a supervisor dashboard, adaptive per-crew baselines for rotating workers: all of these are legitimate, all of them were tempting to claim as "built," and all of them are Phase 2. Saying that explicitly, in writing, instead of quietly implying a closed loop that doesn't exist yet, took more discipline than building any individual segment.

Accomplishments That We're Proud Of

A fully specified, graceful-degradation verification chain in Segment 1 that never dead-ends: it always resolves to a pass, a fallback check, or an explicit unresolved handoff.
Replacing a hardware-dependent hazard-positioning plan with one built entirely from what a worker is already wearing, without losing the core safety behavior.
A genuinely deterministic, auditable escalation layer that we can explain end-to-end to a non-technical judge without hand-waving "the AI decides."
Being willing to write "Phase 2, not built" next to features that would have made the pitch sound bigger, and betting that judges reward that honesty more than they reward an implied capability that isn't real.

What We Learned

The most defensible AI systems are hybrids, not end-to-end models. Deciding where to put a model and where to put a rule turned out to matter more than which model we picked.
Scope cuts are design decisions, not compromises, when they're made explicitly and for a stated reason: dropping beacon hardware for glasses-heading positioning made the system more honest, not less capable on paper.
A confidence threshold is not a finished feature until you've designed what happens below it. We learned this the hard way after our first verification chain had nowhere to go on a low-confidence frame.
The best answer to "how does it predict X" is often the simplest one you can defend. A moving average we can explain beats a "forecasting model" we'd struggle to justify under questioning.