Thumbnail
Autonomous Email sent to
Analysing the KPIs for the order
Scanning the products delivered
Demo_Portal

DRISHTI: AI Site Ledger Through Meta Smart Glasses

Tagline: Kaya knows what should arrive. DRISHTI verifies what actually did hands-free, at the site gate.

Track: Physical AI · Team: Vedant, Anant, Kartikya, Harshit (IIT Bombay) · Kaya AI Hackathon 2026

Inspiration

Construction runs on paper the moment it leaves the software. ERPs, POs, BOQs, and drawings all know what should happen but the instant material reaches the site gate, gets stored, issued, and installed, the digital record goes blind. Every dispute starts with the same question: what actually happened on site?

Talking to people who live this a warehouse operations lead managing 3 stores and coordinating with 100+ one pattern kept repeating: every verification step that depends on a human remembering to scan silently fails under load. Materials are ~40% of a construction budget, rework runs 2–20% of contract value, and 1–2% of material value quietly leaks at receiving, issue, and install.

Kaya already tracks materials from submittal to last-mile delivery. We asked: what's the missing link? It's the last 100 feet the physical moment a delivery crosses the gate. That's a perception problem, and it's exactly what wearable Physical AI is for.

Our core insight: don't put AI glasses on the 200 workers who build the site put them on the ~5 people who run material flow (storekeeper, site engineer, QA/QC, supervisor). One ₹30k device on the right person verifies hundreds of material movements a shift. That's economics worker-wide deployment can never match.

What it does

DRISHTI turns Ray-Ban Meta smart glasses into a voice-first "Smart GRN" (Goods Receipt Note) layer:

A truck arrives. The storekeeper says "Start GRN for Tower A cement delivery."
The glasses read the delivery challan (OCR) and pull PO, item, grade, quantity, batch, EAN.
They scan the load counting bags, flagging damaged and wrong-SKU units with confidence scores, grounded to a QR batch anchor.
Observed facts are matched against the Kaya PO + contractor ERP/WMS context.
DRISHTI speaks the exception back: "PO expects 500. I count 486–492. 3 damaged. Create GRN with exception?"
One voice confirmation writes a signed, append-only ledger event with timestamped POV evidence.
Kaya's agent Jarvis then emails the supplier the proof and schedules reconciliation no human drafting.

Crucially, the ledger cuts both ways: it protects the buyer from short/damaged/wrong deliveries and exonerates honest suppliers when material arrived correct. It replaces "your word against mine" with one timestamped truth which is what makes it adoptable.

The same event ledger then extends, with zero new hardware, to three-way invoice matching, hidden-work evidence, QA/QC, and auto-DPR.

How we built it

Demo (what's live in-browser): a control-room web app with a POV detection console, real challan OCR (Tesseract.js), deterministic PO matching, a cryptographically-signed event ledger (JSON with an Ed25519/SHA-256-style signature), a live "₹ protected" impact counter, an Operations Center (exception queue, leakage-by-supplier, reason breakdown), and actual voice output via the Web Speech API so the audio-first UX is demonstrated, not just described. Frontend in vanilla HTML/CSS/JS with a dark, AR-HUD-inspired aesthetic.

Proposed production stack (architected, edge-deployable today):

Capture: Ray-Ban Meta Gen 2 (12 MP / 3K) via the Meta Wearables Device Access Toolkit → Android phone gateway.
Detection & counting: YOLOv11-n / RT-DETR (INT8, TensorRT), ByteTrack / OC-SORT tracking so events aren't double-counted, a density head for dense stacks, and QR batch anchors for identity.
Reasoning: a compact 3–4B VLM (Qwen2.5-VL / Florence-2 / Moondream) at 4-bit (AWQ/GGUF) via TensorRT-LLM for damage/wrong-brand/spec checks.
Documents: Donut / PaliGemma-2 doc-VLM + Surya / PaddleOCR (Indic) fallback.
Voice (Hinglish): AI4Bharat IndicConformer / IndicWhisper + a grammar-constrained intent parser + Piper TTS.
Matching: deterministic rules + LightGBM, temperature-scaled probabilities with a cost-of-error threshold.
Edge: NVIDIA Jetson Orin Nano Super (67 TOPS), DeepStream, sub-2s p95 feedback, offline-first sync.

Architecture principle: a system of observation (glasses + edge) bound to a system of record (Kaya + ERP), with a human confirming every money-moving action.

Challenges we ran into

Counting the same bag twice. Naive frame-by-frame counting double-counts every time the wearer glances back. We designed an IMU-gated cascade (pose gate → sampler → cheap detector → VLM only on survivors) plus multi-object tracking and QR batch anchors so we count events, not frames.
Being honest about today's hardware. The Meta DAT is phone-mediated and preview-stage — no on-lens compute or AR overlay. We deliberately scoped to what ships now: 12 MP stills, streaming, voice, and audio confirmations, with all perception off-glass. The load-bearing capability is a good camera + mic routed to real compute, not future AR.
Trust and calibration. Raw model scores aren't trustworthy probabilities, so we added temperature scaling (minimizing ECE) and a cost-of-error rule high-value or safety-critical items get stricter thresholds and human review.
Bulk material honesty. You can't count sand. We scoped the MVP to packaged, labeled SKUs and handle bulk via challan + weighbridge + e-way-bill cross-checks instead of pretending to count.
Indian-site reality. Handwritten challans, Hinglish code-switching, noisy gates, and unreliable connectivity drove offline-first design and Indic ASR choices.

Accomplishments that we're proud of

A working end-to-end demo real OCR → deterministic match → signed ledger → Jarvis draft → impact counter that a judge can drive live, plus a synced 2-minute auto-narrated showpiece with real voice.
A sharp, defensible wedge: instrument 5 roles, not 200 workers with unit economics that pay back a per-site deployment inside the first month (~₹3 lakh/month protected vs ~₹1–1.5 lakh capex).
A design that fits Kaya natively the physical sensing front-end for the last 100 feet, feeding Jarvis verified evidence no procurement-only competitor has.
Technical depth that survives scrutiny: a concrete, current, edge-deployable CV/ML stack rather than hand-waving.
A product that's India-first, not India-only the identical layer applies to Kaya's US data-center and mission-critical projects, where one mis-received switchgear delivery costs more than a whole site's deployment.

What we learned

The moat is context, not vision. Generic glasses can say "this is a cement bag." Kaya-powered DRISHTI can say "this is the wrong grade for PO-1042, bound for Tower B, not Tower A." Construction isn't an image-captioning problem it's a context-matching problem.
Physical AI wins by being present, not smart. The value isn't a bigger model; it's being at the moment of receiving instead of behind a dashboard waiting for late data entry.
Honesty is a feature. Scoping out bulk counting, naming exact hardware limits, and keeping humans in the loop made the proposal more credible, not less.
Audio-first beats AR overlays for a moving worker with full hands concise spoken prompts and repeat-back confirmations are the right UX.

What's next for DRISHTI

Phase 0 → 1: table-top demo → warehouse/site-store pilot (2–3 storekeepers) measuring GRN time, scan compliance, and exception evidence completeness.
Phase 2: live construction store with real PO + contractor BOQ/work-order context and human-confirmed stock updates.
Phase 3: site-round expansion wrong-issue verification, QA/QC pass-fail proof, hidden-work (pre-pour rebar) evidence, and auto-DPR from verified events.
Scale path: single store → multi-store warehouse → construction site → project portfolio → predictive ops (repeat-offender suppliers, common wrong-issue SKUs, contractor quality trends).
Bridge to Kaya's core market: the same verified-receiving layer on US data-center and mission-critical builds.