SiteGuard

AI safety agent for construction sites, detects OSHA violations in real-time from video feeds and answers compliance questions instantly.

Comment

Inspiration

My dad has spent 3 decades on Fertilizer manafucturing site. He's a safety supervisor at SSR Mechanical Maintenance, the kind of person who walks a site and just knows when something is off before he even sees it. Growing up watching him come home exhausted, knowing he'd spent the day fighting to keep people alive, that stuck with me. It gave me a deep respect for what safety actually looks like on the ground.

When I started my mechanical engineering internships, I finally saw it for myself. Three different sites, three different companies, same story everywhere. A worker climbs without a harness. An inspector walks past. Nobody catches it. Not because anyone was careless, but because one person physically cannot watch everything, everywhere, all the time.

And the audit process? Still PowerPoints. Still paper checklists. Still one person with a clipboard trying to document violations faster than they can be forgotten.

This isn't some edge case. Every year, thousands of workers in construction, oil and gas, and mechanical maintenance don't come home. Not because the work is impossible to do safely, but because the violation that caused the incident was never caught in time.

$$ \text{Construction} \approx 1 \text{ fatality in every } 5 \text{ worker deaths across all US industries} $$

Over 400,000 serious injuries happen annually across construction, oil and gas, and mechanical maintenance. Burns. Falls. Crush injuries. Amputations. Behind almost every single one, investigators find the same root cause: a safety violation nobody caught.

I kept thinking, we have AI that can recognize faces in a crowd, but we can't tell when a worker isn't wearing a helmet? That question became SiteGuard AI.

What it does

SiteGuard AI is an always-on safety agent for construction and industrial sites. You upload a site video and Gemini 2.5 Flash analyzes every frame, identifies OSHA violations, and streams findings to a dashboard in real time.

But the part that matters most to me isn't the known violations, the missing hard hats and blocked exits. It's what the system can discover on its own. A vision model trained on the breadth of human knowledge doesn't just know the checklist. It can flag something a human inspector might not even recognize as dangerous: an unusual pipe configuration, a chemical stored too close to something it shouldn't be near, a non-standard equipment setup. Stuff that doesn't have a checkbox yet but is just as real and just as deadly.

The ARIA intelligence agent (built on Google ADK) sits alongside the dashboard and answers questions in plain English:

"What were the most critical violations in today's inspection?" "Which OSHA codes were breached and what corrective actions are needed?"

ARIA isn't just generating text. Google ADK gives it real tool-calling capabilities, so when you ask a question, it actively queries your site data through specialized tools: get_violations, get_recent_violations, search_osha_standards, and get_live_site_status. Ask "what violations were found in this inspection?" and ARIA responds in under 200 milliseconds. That speed comes from the fast-path architecture I built. For common queries, ARIA skips the full agent reasoning loop and pulls data straight from Firestore and BigQuery. For complex questions, it activates the full ADK pipeline and chains multiple tools together to build a detailed answer.

Evidence images show up inline inside the chat, so safety managers can verify every detection without leaving the conversation. Everything is scoped to the current inspection session, ARIA only sees data from this scan, keeping results clean and relevant. And it's available on every page, Inspect, Dashboard, and Reports, each with context-aware preset questions tailored to what you're looking at.

Auto-generated HSE compliance reports map every violation to its OSHA standard, embed evidence images, and are ready to download. What used to take hours of manual documentation takes a single click.

How I built it

The whole stack runs on Google Cloud and Gemini:

  • Gemini 2.5 Flash analyzes video frames at 0.5fps through Vertex AI. Each frame returns structured violation data mapped to OSHA 29 CFR Part 1926 standards.
  • Gemini Live API (gemini-live-2.5-flash-native-audio) powers Live Supervisor mode with real-time voice and vision plus native barge-in support.
  • Google ADK LlmAgent powers ARIA with function tools: get_violations, get_recent_violations, get_live_site_status, and search_osha_standards. The ADK agent handles tool selection, chaining, and response synthesis. For common queries, a keyword-matching fast path bypasses the full agent loop and hits Firestore directly, getting responses under 200ms. Complex questions go through the full ADK pipeline.
  • Cloud Firestore stores violations, sessions, cameras, and alerts in real time.
  • BigQuery handles 30-day analytics, risk scoring, and compliance trends.
  • Cloud Storage holds evidence frame images, recordings, and generated reports.
  • Cloud Run hosts both the FastAPI backend and React frontend as separate containers.
  • Secret Manager keeps all credentials out of code and Docker images.

The frontend is React 18 + Vite + Tailwind CSS v4 + shadcn/ui. The backend is Python 3.12 + FastAPI with Server-Sent Events for the live violation stream and WebSockets for ARIA and the supervisor agent.

I built smart deduplication to prevent the same hazard from being flagged on every single frame. It works on a sliding window of 20 frames across canonical hazard categories:

$$ \text{flag} = \begin{cases} \text{true} & \text{if } \nexists\ v_i \in W_{20} \text{ s.t. } \text{category}(v_i) = \text{category}(v) \ \text{false} & \text{otherwise} \end{cases} $$

In plain terms: if the same category of violation already appeared in the last 20 frames, skip it. Otherwise, it gets flagged.

Challenges I ran into

Real-time pipeline at scale. Ten minutes of video is 300 frames. Each one needs a Gemini API call, frame annotation, Firestore write, and SSE push to the browser, all happening at the same time. Getting that to feel live without overwhelming the backend took careful async design throughout.

Deduplication across model output. Gemini describes the same hazard differently across frames. "Worker without helmet" in one frame becomes "no head protection observed" in the next. Raw string matching failed completely. The fix was mapping all output to a canonical HazardCategory enum before deduplication, not after.

ARIA session isolation. ARIA needed to know which inspection to answer about. I implemented a set_inspection_session WebSocket control message that scopes every subsequent query to the active session. Without this, ARIA would mix data across different sites and scans.

Fast path vs. full agent latency. The full ADK agent takes 2 to 5 seconds. For common queries I built a keyword-matching fast path that hits Firestore directly and returns in under 200ms. Routing between the two paths transparently gave users both speed and intelligence.

React hooks ordering. Temporal Dead Zone errors surfaced when useCallback hooks referenced values from hooks declared later in the component. Invisible until runtime. The fix: always declare provider hooks before consumer hooks. Simple in hindsight, painful to diagnose.

Accomplishments I'm proud of

I built a complete end-to-end safety pipeline: video in, OSHA-mapped violations out, streaming live to a browser, deployed on Google Cloud Run. All solo.

ARIA answers real natural-language safety questions backed by live Firestore and BigQuery data. These aren't hardcoded responses.

The auto-generated HSE compliance reports would replace hours of manual documentation, with evidence images and corrective actions included.

And the thing I keep coming back to: the system can find violations nobody thought to look for. It's not just checking a list. It actually understands a scene.

What I learned

The hardest part of AI on a real worksite isn't the model. It's everything around it. Gemini is remarkably good at seeing hazards. The real engineering challenge was building the session management, routing logic, latency optimizations, and data pipelines that make those detections actually useful to a safety supervisor who's under pressure and doesn't have time to wait.

I also learned that speed matters as much as accuracy in safety tooling. A 5-second response is fine for generating a report. It's way too slow when a supervisor is asking "is this safe right now?" Building two separate paths, one fast and one intelligent, and routing between them transparently was one of the most important architectural decisions I made.

And honestly, the biggest lesson: safety technology only works if the people on the ground trust it and use it. The UI, the ARIA conversational interface, the plain-English reports, all of that exists because a tool that sits unused doesn't save anyone.

What's next for SiteGuard

The violations I can detect today are the ones Gemini can see in a single frame. Here's what I want to build next:

  • Temporal hazard detection - patterns that only show up across multiple frames: a worker gradually fatiguing, a structural element shifting over hours, unsafe workflows that look fine in isolation
  • Unknown violation discovery - using Gemini's broader knowledge to flag hazards that don't exist on any current checklist, then feeding confirmed discoveries back into the OSHA knowledge base
  • Multilingual worker alerts - the Live API supports 70+ languages; I want real-time voice warnings delivered directly to workers in their own language, not just reports sent up to supervisors
  • Oil and gas and mechanical maintenance profiles - specialized violation models for high-pressure systems, lockout/tagout, confined space entry, and the hazards specific to industries where my dad has spent his career
  • Offline edge deployment - sites in remote locations with no reliable internet need local inference; taking the violation detection pipeline to the edge is the next infrastructure challenge

Every one of those features exists because somewhere, right now, there's a worker on a site where something dangerous is happening and nobody has caught it yet. That's what keeps this project moving.

Built With

  • bigquery
  • docker
  • gemini-live-api
  • google-agent-development-kit-(adk)
  • google-cloud-build
  • google-cloud-storage-gemini-2.5-flash-api
  • google-generative-ai-sdk-websocket
  • google-secret-manager-cloud-firestore
  • javascript
  • opencv
  • python
  • radix-ui
  • react
  • react-router-google-cloud-run
  • recharts
  • rest-ffmpeg
  • server-sent-events-(sse)
  • shadcn/ui
  • tailwind-css
  • typescript-fastapi
  • uvicorn
  • vite
  • webrtc
Share this project:

Updates

Submission history