Kaya Ai

Inspiration

Construction sites remain one of the most dangerous workplaces in the world. Workers constantly face risks such as missing safety equipment, exposed electrical wiring, unguarded edges, falling objects, and fire hazards. Existing AI assistants only respond when someone asks a question, but during an emergency, workers often don't have the time or ability to interact with an assistant.

This inspired us to ask a simple question:

What if AI could continuously see what the worker sees and warn them before an accident happens?

Instead of creating another chatbot, we wanted to build an AI Safety Supervisor that actively monitors the environment, detects hazards in real time, speaks to workers without being prompted, and helps supervisors monitor the entire construction site from a live dashboard.

That idea became KAYA AI.

What it does

KAYA AI is a real-time industrial safety platform powered by AI, smart glasses, and live voice interaction.

Workers wear smart glasses (or a smartphone during development), allowing the AI to continuously receive live camera and microphone streams.

The platform can:

Continuously monitor the work environment using AI vision Detect safety hazards such as: Missing helmets Unsafe edges Trip hazards Exposed wiring Smoke and fire Interrupt workers immediately with spoken safety warnings Draw visual hazard bounding boxes on the worker's display Log incidents automatically with photographic evidence Answer blueprint and structural specification questions hands-free Search uploaded safety documents using voice Recognize products and inventory through AI vision Store inventory location and retrieve it later Allow supervisors to monitor workers from a live dashboard Display live worker conversations, incidents, inventory, and documents Maintain persistent worker identity across reconnects

Unlike traditional assistants, KAYA AI is proactive - it doesn't wait for the worker to ask for help before responding.

How we built it

The system follows a cloud-based real-time architecture connecting workers, supervisors, and AI through WebSockets.

Backend Python 3.11 FastAPI Uvicorn SQLite WebSockets AI Google Gemini Live API Google Agent Development Kit (ADK) Real-time Voice + Vision Lightweight vision model for hazard detection Retrieval-Augmented Generation (RAG) for blueprint and document search Frontend

Worker Device

Android application Kotlin Jetpack Compose CameraX OkHttp WebSockets

Supervisor Dashboard

HTML CSS JavaScript Live monitoring interface Infrastructure Docker Google Cloud Run GitHub Actions Cloud APK build pipeline Core Architecture

Our biggest innovation is the Safety Sentinel, a background AI process that continuously analyzes incoming camera frames.

Instead of waiting for user input, the Safety Sentinel independently detects hazards and injects alert messages into the live Gemini session. This causes the AI to interrupt the worker with an immediate voice warning while simultaneously logging the incident and notifying supervisors.

This proactive architecture transforms a passive AI assistant into an active safety supervisor.

Challenges we ran into

Building a real-time industrial AI system presented several technical challenges.

Real-time Voice + Vision

Streaming camera frames, microphone audio, AI responses, and synthesized speech simultaneously required reliable low-latency communication over WebSockets.

Making AI Speak First

Most conversational AI systems only respond after receiving user input.

Designing the Safety Sentinel to independently analyze camera frames and inject alerts into the live AI session required a completely different architecture than a traditional chatbot.

Live Tool Calling

Initially, some live AI tool calls experienced delays because of model selection issues.

We resolved this by selecting the appropriate real-time Gemini model for immediate tool execution.

Stable Worker Identity

Workers frequently disconnect because of unstable construction-site networks.

We implemented persistent worker identities so reconnecting devices automatically restore previous profiles and dashboard information.

Reliable Vision Processing

Vision models occasionally fail to extract every product detail.

Instead of losing information, KAYA always stores the original image so supervisors can review it later.

Accomplishments that we're proud of

Some achievements we're most proud of include:

Built a fully functional real-time AI safety assistant Implemented proactive AI that warns workers before being asked Developed the Safety Sentinel architecture Integrated real-time voice and vision into one AI workflow Created automatic hazard logging with image evidence Built a live supervisor dashboard Implemented blueprint retrieval using RAG Added AI-powered product and inventory recognition Built persistent worker identity across reconnects Successfully containerized and deployed the backend on Google Cloud Run Automated Android APK generation using GitHub Actions

Most importantly, we transformed AI from a reactive assistant into an active industrial safety supervisor.

What we learned

During this project we learned much more than simply integrating AI APIs.

We learned:

Designing production-grade real-time AI systems Managing live bidirectional media streams Building scalable WebSocket architectures Using Google Gemini Live API effectively Integrating Google ADK with FastAPI Applying Retrieval-Augmented Generation for industrial documentation Building proactive AI workflows instead of reactive assistants Designing AI systems that prioritize safety and human interaction Creating reliable cloud deployments with Docker and Cloud Run

Perhaps our biggest takeaway was that the true value of AI is not just answering questions—it is recognizing situations where people need help before they ask.

What's next for Kaya Ai

Our roadmap focuses on turning KAYA AI into a production-ready industrial safety platform.

Future improvements include:

Native Meta Smart Glasses integration On-device AI inference for lower latency Multi-site enterprise dashboard OSHA-compliant automatic incident reports Offline operation with synchronization Predictive safety analytics using historical incidents AI-generated daily and weekly safety reports Computer vision for additional PPE detection Integration with enterprise ERP and inventory systems Support for multiple languages Voice biometrics for worker authentication Digital twin integration for construction sites Fleet management across multiple industrial locations

Our long-term vision is to make KAYA AI the always-present AI Safety Supervisor that helps prevent workplace accidents before they happen, making construction sites safer, smarter, and more connected.