WITNESS

Construction sites forget. Witness remembers — smart glasses that catch the wrong part before the concrete.

Team Espada · IIT Hyderabad · Kaya AI IIT India Hackathon 2026 · Track 1 — Physical AI


Inspiration

Kaya already tracks a construction project's whole supply chain — submittal, procurement, delivery. But we kept circling one gap it can't see: the last mile, the moment a worker on a beam actually installs the part. The approved revision might have changed three weeks ago, and nobody up there knows.

The number that hooked us was rework. Studies from CII and McKinsey put it at 5–15% of total project cost, and a meaningful slice of that is nothing more exotic than the wrong version of the right part going in. The approved answer existed the whole time — it just never reached the person holding the wrench. That's not an intelligence problem. It's a verification problem. And a pair of smart glasses is standing in exactly the right place to solve it.

What it does

Witness runs on Meta smart glasses and does three things at the moment of install — Ask, Verify, Remember:

  • Ask — hands-free, it surfaces what's currently approved for this location: "GT-12, Rev C — approved here."
  • Verify — one tap starts a scan of the tag the supply chain already prints on the part. Witness checks that scanned unit against what's approved for this spot and flags a mismatch: scanned Rev B, approved Rev C. An NCR drafts itself; a human signs off.
  • Remember — every verified install becomes a lesson. Three weeks later, when another worker reaches for the same wrong part, Witness speaks up: "GT-12 caused three mismatches in Zone A — double-check before installing."

Zoom out and the site becomes a live map: a rework heat-map by zone, a ranked list of the most-confused components, and a supervisor alert when a hotspot emerges. It's institutional memory for a jobsite — not a chatbot.

How we built it

Architecture is glasses as the interface, phone as the edge, cloud for reasoning:

  • Glasses — the worn interface. POV camera + open-ear audio via Meta's Wearables Device Access Toolkit (developer preview). A tap starts capture (push-to-talk — the SDK doesn't expose a custom wake word yet, so we designed with that constraint instead of against it).
  • Phone — the paired edge the glasses SDK routes through. Decodes the tag and runs the first mismatch check on the spot, so a weak signal or a dusty tag doesn't stop the check.
  • Cloud — an AI model doing RAG grounded in Kaya's approved submittal record. When it isn't sure, it asks a human. The reasoning layer is model-agnostic — no single-vendor lock-in.
  • Graph — verified installs and error patterns write back, feeding the dashboards, the heat-map, and the next warning.

For the submission itself we built a 10-slide deck, a ~2-minute animated explainer (a from-scratch React timeline engine — Stage / Sprite / Camera with deterministic track() keyframes, a hand-drawn jointed-figure rig, a first-person AR HUD, and a burned-in VO track), and a full voice-and-cut production guide.

Challenges we ran into

  • The tag→revision trap. Our first cut implied the scan reads a revision off the part. It usually can't — a code resolves identity/SKU, and a revision is often just paperwork on an identical unit. We split the job: Ask surfaces the approved revision; Verify confirms the right item for this location and catches units that map to a superseded rev. That single reframe made the whole pitch defensible.
  • Adoption, not tech. A piece-rate crew won't scan if it's extra work. We reframed the scan as the field-verification sign-off that submittal-controlled equipment already requires — so using Witness replaces a step instead of adding one.
  • The wake-word reality. Meta's SDK is push-to-talk today. Rather than promise magic, we made the tap a visible, intentional beat in the demo.
  • Economics that survive a skeptic. We dropped an unsourced "≈half of rework is version-linked" and rebuilt on a deliberately conservative slice (below), because a number nobody can attack beats a bigger one someone can.
  • Grilling our own idea. We spent a full pass attacking Witness as if we wanted it to fail — the tag question, adoption, "why not just Kaya," liability. Most of what's good in the final version came out of that pass.

Accomplishments that we're proud of

  • A crisp scoped-by-design story: Witness confirms identity, location and superseded revisions; for paperwork-only changes it surfaces the approved rev rather than guessing; it advises and drafts, a human confirms — never workmanship, never a green-light on compliance.
  • Honest, defensible economics. On a \$10M project:

$$ \text{Rework} = 0.05\text{–}0.15 \times \$10\text{M} = \$0.5\text{–}1.5\text{M} $$

$$ \text{Wrong-item slice} \approx 0.15 \times \text{Rework} = \$75\text{–}225\text{K} $$

$$ \text{Saved} = \tfrac{1}{3} \times \$75\text{–}225\text{K} = \$25\text{–}75\text{K} \;\Rightarrow\; \text{one prevented tear-out pays for every pair of glasses on site.} $$

  • A from-scratch animation engine — every frame a pure function of time, so the film is fully deterministic and re-renderable — plus an AR POV that makes the feasibility criterion visible on screen.
  • A pitch and a film that tell the identical, defensible story, right down to the opening line.

What we learned

  • Physical AI is a placement problem as much as a model problem. The intelligence to prevent this rework already existed; the win is putting it in front of the right person at the right second.
  • Scoping is a feature. Saying clearly what Witness doesn't do — read workmanship, decide compliance, detect a paperwork-only rev by photo — made it more convincing, not less.
  • Build the defensible thing, and it's usually the buildable thing too. "Look up what's approved here and compare identity" is a normal RAG task; "detect a revision from a photo" is a research project. Honesty and feasibility pointed the same way.
  • Design with the platform's constraints, not around them. Push-to-talk stopped being a limitation the moment we made it part of the interaction.

What's next for Witness

  • Now — concept, deck, and animated explainer (this submission).
  • Next — a single-zone pilot: ~200 installs, measured on three falsifiable metrics — catch rate (% of rev mismatches flagged), time-to-answer (< 10s), and coverage (% of installs field-verified). The scan-plus-cloud core is buildable today; the pilot ships to test users on Meta's live SDK.
  • Then — memory at scale: cross-project error transfer, offline-first at the edge, and on-prem for enterprise sites. Every site Witness touches makes the next one smarter.

Kaya sees the whole supply chain. Witness closes the last blind spot — the moment materials become reality.

Built With

  • capcut
  • elevenlabs
  • graph-database
  • javascript
  • meta-wearables-device-access-toolkit
  • obs
  • python
  • python-pptx
  • qr-barcode-scanning
  • rag
  • ray-ban-meta
  • react
  • svg
  • vector-database
Share this project:

Updates

Submission history