-
-
Live Dashboard
-
Architecture
Inspiration
Construction is one of the world's largest yet least digitized industries. Despite employing millions of workers, supervisors still rely heavily on manual observation, paper drawings, and radio communication to manage safety and operations. A single supervisor cannot monitor an entire construction site simultaneously, and workers often lose valuable time stopping work just to access project information.
We asked ourselves a simple question:
What if every worker became the eyes of one intelligent AI capable of supervising the entire construction site?
That idea became Sentinel.
What it does
Sentinel is an Agentic AI Construction Operating System that connects every worker's smart wearable to a centralized Agentic AI Central Brain.
Each worker's wearable runs an Edge Vision-Language Model (VLM) that continuously understands its surroundings by detecting PPE violations, hazards, equipment, and worker activities.
Instead of treating every wearable as an isolated device, Sentinel combines observations from every worker into one shared intelligence that can:
- Detect PPE violations and hazards in real time
- Escalate repeated safety issues to supervisors
- Answer workers' questions using Retrieval-Augmented Generation (RAG) grounded in project documents
- Build dynamic danger zones across the construction site
- Provide supervisors with a live operational dashboard
- Coordinate multiple workers simultaneously
Every validated observation is stored inside a dedicated Construction Intelligence Database, creating a continuously growing construction-specific dataset that can be used to improve future Edge Vision-Language Models.
How we built it
For the hackathon MVP, Sentinel is built around Meta Smart Glasses, following the hardware provided in Kaya AI's Physical AI track.
The architecture consists of four major components:
Edge Perception Layer
Each wearable captures the worker's perspective and performs environmental understanding using Vision-Language Models.
Agentic AI Central Brain
A centralized AI orchestrator dynamically routes incoming events through specialized pipelines, including:
- Safety Monitoring Pipeline
- RAG Knowledge Pipeline
- Danger Zone Pipeline
- Supervisor Escalation Pipeline
- Construction Intelligence Pipeline
Construction Intelligence Database
The Central Brain stores:
- Structured observations
- Contextual image snapshots
- Supervisor validations
- Project documents
- Construction metadata
This continuously growing dataset is used to periodically evaluate and fine-tune future Edge Vision-Language Models.
Supervisor Dashboard
The dashboard provides a live digital twin of the construction site, displaying:
- Worker locations
- Dynamic danger zones
- Active hazards
- AI pipeline activity
- Construction intelligence
- Site-wide analytics
Challenges we ran into
One of our biggest challenges was balancing an ambitious long-term vision with a realistic hackathon MVP.
Current consumer smart glasses do not have enough compute to run advanced Vision-Language Models entirely on-device, so we separated edge perception from centralized reasoning.
Another challenge was designing a system that continuously improves without storing continuous raw video. Instead, we focused on structured observations, contextual image snapshots, and supervisor validation to build a scalable and privacy-conscious Construction Intelligence Database.
Finally, integrating computer vision, RAG, edge AI, dynamic danger zones, and multi-worker coordination into one coherent architecture required multiple design iterations before arriving at our current Agentic AI approach.
Accomplishments that we're proud of
- Designed an end-to-end architecture for an Agentic AI Construction Operating System.
- Created the concept of a centralized Agentic AI Central Brain capable of coordinating an entire construction site.
- Designed a futuristic supervisor dashboard that visualizes live construction intelligence.
- Proposed a continuous learning pipeline where supervisor-validated observations become construction-specific training data.
- Built a scalable roadmap from today's smart wearables to tomorrow's purpose-built AI construction hardware.
What we learned
Building Sentinel taught us that solving construction problems requires much more than object detection.
Real construction intelligence combines:
- Edge AI
- Computer Vision
- Vision-Language Models
- Retrieval-Augmented Generation
- Agentic AI
- Human-in-the-loop validation
- Distributed systems
Most importantly, we realized that the real innovation is not the wearable itself—it is the Agentic AI Central Brain that combines every worker's observations into one continuously improving construction intelligence platform.
What's next for Sentinel
Our long-term vision is to evolve Sentinel into the AI operating system for construction.
Future work includes:
- Building custom AI wearables capable of running Edge Vision-Language Models directly on-device.
- Expanding the Construction Intelligence Database with supervisor-validated observations and contextual images from real construction sites.
- Continuously fine-tuning construction-specific Vision-Language Models to improve perception accuracy across different projects.
- Automatically generating live site maps and dynamic danger zones from worker observations.
- Monitoring construction workflows to detect incorrect task sequences before they become costly rework.
- Integrating predictive analytics, scheduling, procurement, and site intelligence into the Agentic AI Central Brain.
Our ultimate goal is to build an AI Construction Supervisor capable of understanding, coordinating, and continuously improving construction sites at scale.
One AI. Many eyes. A construction site that continuously learns, understands, and improves itself.
Log in or sign up for Devpost to join the conversation.