NETRA — The Site's Eyes

Hands-free AI on Ray-Ban Meta glasses for construction's last 50 feet: receives deliveries at the gate, answers specs in Hindi, and whispers IS-code-grounded hazard warnings from the edge.

Comment

Inspiration

Construction's biggest failures are supply-chain failures: the "$1M-per-day problem." Reading practitioner threads, one receiving lead's words stuck with us: "the packing slip is distributed to the teams days later via email… It's a terrible system." (r/Construction). The failure concentrates in the last 50 feet of the supply chain: the site gate, where deliveries meet paper challans and data goes to die. Bad data may have cost global construction up to $1.85 trillion in a single year (Autodesk/FMI), workers burn 35% of their time hunting for information (FMI/PlanGrid), and India loses 38 construction workers every day (British Safety Council/IIT Delhi).

Our insight: a worker's hands are the least available input device on a site and their eyes are its most valuable unused sensor. Ray-Ban Meta glasses finally make that sensor programmable.

What it does

NETRA (Sanskrit: "eye") is a hands-free AI teammate on Ray-Ban Meta smart glasses. Three verbs:

  • Receive: look at a pallet: NETRA reads labels and packing slips, matches them against the PO and submittal register, counts, flags shortages by voice, and logs geotagged proof to the project graph. Zero day-of surprises.
  • Ask: "Netra, what's the torque spec for these anchor bolts?" Voice answers drawn from drawings, specs, and submittals in Hindi, Tamil, or English. No new UI; workers just talk.
  • Guard: an ambient safety layer that reasons about the scene (not another PPE camera) and whispers warnings grounded in the actual IS-code clause.

Every receiving event feeds a probabilistic lead-time model (Chronos-2) that predicts ROJ slippage weeks before it bites. NETRA moves work, it doesn't just report on it.

How we're building it

Edge-first, because site connectivity dies: glasses → phone (Meta Wearables Device Access Toolkit) → NVIDIA Jetson. A YOLO detector runs at full frame rate and gates a small VLM (Qwen3-VL-4B, LoRA fine-tuned on the ConstructionSite-10k dataset) so real-time scene understanding fits an edge power budget (arXiv 2604.05210, arXiv 2508.11011). Safety callouts use regulation-RAG over IS codes, if it isn't retrieved, it isn't said (arXiv 2510.04145). Voice runs on streaming ASR + AI4Bharat Indic TTS across 22 languages.

Challenges and how we de-risked them

  • No display on Gen 2 glasses → voice-first is the honest architecture, not a workaround.
  • Hardware arrives only after shortlisting → our Meta Wearables developer org is already live, and the DAT SDK's MockDeviceKit lets us build and test the full glasses pipeline before devices arrive.
  • Site connectivity → all inference on the edge; the project graph syncs when the network returns.

What we learned

Zero-shot VLMs demonstrably miss construction hazards. Fine-tuning with published before/after evals is non-negotiable. And the highest-leverage AI on a site isn't a dashboard; it's an agent standing at the gate.

What's next (Stage 2, Jul 17–31)

Week 1 on MockDeviceKit: YOLO gate + VLM fine-tune with its eval table, done by day 5. Week 2 on hardware: an end-to-end staged-gate demo, real pallet, real packing slip, Hindi voice query, live hazard callout plus a full backup tape.

Team: Ajayaditya Lokchandra (BS Data Science & Applications, IIT Madras + BE Mechatronics, KCT) · Nithisha Venkatesh (BTech AI & Data Science, KCT)

Built With

  • ai4bharat
  • android
  • chronos-2
  • edge-computing
  • indic-tts
  • kotlin
  • lora
  • meta-wearables-dat
  • nvidia-jetson
  • opencv
  • or-tools
  • pipecat
  • python
  • pytorch
  • qwen3-vl
  • rag
  • ray-ban-meta-glasses
  • streaming-asr
  • tensorrt
  • ultralytics
  • yolo
Share this project:

Updates