SiteSense AI

Project Name: SiteSense AI Tagline: Every worker becomes a sensor. Every site becomes intelligent. Track: Physical AI

Elevator Pitch: 38 construction workers die every day in India. Not because rules don't exist, but because of a massive language and knowledge barrier. SiteSense AI is a multi-modal field intelligence platform running on Meta Smart Glasses that sees hazards, hears equipment failures, translates instructions into 8 Indian languages in real-time, and captures tacit expert knowledge before it retires.

The Problem: India's construction sector loses thousands of lives and billions of dollars to three core problems:

The Language Barrier: Migrant workers speak Bhojpuri, Tamil, or Odia, but instructions are given in Hindi or English. Written warnings are useless to a workforce with 81% illiteracy. Sensory Overload: Workers process 90dB noise and extreme heat, making them blind to surrounding hazards. Knowledge Drain: When an expert mason with 30 years of experience retires, all their tacit knowledge (how concrete sounds before it cracks) disappears. What SiteSense AI Does: SiteSense acts as the central nervous system of the construction site through 4 core modules:

MultiSense (Vision + Audio): Uses Vision-Language Models (VLM) to detect missing PPE and verify materials against purchase orders. Uses audio CNNs to detect anomalous equipment sounds (e.g., crane bearing failure) or worker distress calls through 90dB noise. BabelSite: Real-time translation of construction terminology across 8 Indian languages, delivering translated audio directly to the worker's ear via smart glasses. ShadowMaster: Records the point-of-view and voice narration of expert workers to build an RAG-powered knowledge base. Novice workers can ask "Amber" (our AI) for contextual tips. ClashGuard & ExposureTrack: Monitors cumulative heat/noise exposure and prevents spatial clashes between trades (e.g., plumbers and electricians overlapping). How We Built It (Stage 1 Prototype): For Stage 1, we built a comprehensive web-based command center to simulate the smart glasses experience.

Frontend: Vanilla JS + CSS glassmorphism for a high-performance, dark-mode dashboard with a "Live View" AR simulator. Backend / AI: Built on FastAPI, integrating Llama-3.3-70B (via Groq) for conversational knowledge retrieval (RAG) using Qdrant and fastembed. Visual processing utilizes advanced Vision-Language Models, and translation relies on Indic language models. What's Next for Stage 2: If shortlisted, we will deploy our Python backend onto the NVIDIA Edge Compute module and interface directly with the Meta Smart Glasses camera and microphone arrays. We will transition our simulation into a real-time hardware prototype tested on a live construction site in Bangalore.