Inspiration: Construction sites are among the most complex and dynamic work environments in the world. Every day, project managers coordinate hundreds of workers, dozens of suppliers, and thousands of materials while trying to keep projects on schedule.
Most AI tools today only answer questions after something has already gone wrong. We wanted to build an AI system that continuously observes the physical world, understands what's happening, predicts future problems, and recommends actions before delays or accidents occur. Kaya AI's vision of Amber inspired us to extend that intelligence into the real world through cameras, sensors, and predictive reasoning. Instead of simply reacting to construction issues, Kaya Sentinel proactively prevents them.
What it does: Kaya Sentinel continuously watches the construction site through cameras and sensors.
It detects- Missing helmets Unsafe worker behavior Crane collision risks Unauthorized entry Material shortages It also predicts- Procurement delays Fabrication lead times Ready-On-Job (ROJ) dates Vendor risks Material bottlenecks Finally, it recommends- Procurement actions Alternate suppliers Delivery schedules Resource allocation Safety interventions before problems impact the project.
How we built it: The project consists of four intelligent AI agents coordinated by Amber.
đź‘· Vision & Safety Agent Real-time PPE detection Worker tracking Crane swing detection Restricted zone monitoring Unsafe proximity alerts Built using: YOLOv11 OpenCV Python
Challenges we ran into: One of the biggest challenges was designing a system that combines multiple AI capabilities into one cohesive workflow.
Construction data comes from many different sources including: CCTV cameras Vendor databases Inventory systems Delivery schedules Fabrication reports Synchronizing these data streams while maintaining real-time performance required careful system design. Another challenge was balancing inference speed with computer vision accuracy. Real-time safety alerts demand extremely low latency without sacrificing detection quality. Finally, creating an explainable AI system was essential. Every recommendation made by Sentinel includes the reasoning behind the decision, allowing project managers to trust the AI instead of treating it as a black box.
Accomplishments that we're proud of:
Built a unified Physical AI architecture that combines computer vision, predictive analytics, and supply chain intelligence into a single autonomous system.
- Designed Kaya Sentinel as an extension of Amber, enabling AI to understand real-world construction environments instead of relying only on enterprise data.
- Created a modular multi-agent architecture where Safety, Procurement, Logistics, and Fabrication agents collaborate through shared project context.
- Developed a scalable backend architecture that allows new AI agents and sensors to be integrated with minimal changes.
- Focused on explainability by ensuring every recommendation includes reasoning rather than acting as a black-box AI.
- Built a working MVP that demonstrates how Physical AI can improve worker safety while reducing project delays.
What we learned: During the development of Kaya Sentinel we explored multiple areas of Physical AI.
Some of our biggest learnings included: Building real-time computer vision pipelines for construction environments Detecting PPE violations and hazardous worker proximity Predicting fabrication lead times using historical scheduling data Designing multi-agent AI systems that collaborate instead of operating independently Creating a modular architecture that can easily integrate with Amber Understanding the challenges of noisy real-world construction data Most importantly, we learned that Physical AI is much more than computer vision—it requires perception, reasoning, prediction, and autonomous decision-making.
What's next for Kaya Sentinel:
AR safety glasses for workers Drone-based autonomous inspections BIM integration Live Digital Twin generation Voice interaction with Amber Autonomous procurement approval workflows IoT sensor integration Predictive maintenance for heavy equipment
Built With
- docker
- fastapi
- kayaai(amber)
- networkx
- opencv
- postgresql
- prophet
- python
- react
- streamlit
- xgboost
- yolov11
