SiteSentinel

A pre-shift safety kiosk that catches missing PPE and unfit workers before they walk on the job site, multilingual, edge-AI, $35 of hardware.

Comment

Inspiration

Construction kills more US workers than any other industry. Over 80% of incidents trace back to two preventable causes: missing PPE or a worker who showed up unfit for duty. Today, the only line of defense is a foreman trying to eyeball 50 workers as they file in at 6 AM. That doesn't scale, and people die because of it. We wanted to build the safety check that scales: a thirty-five-dollar device at the gate that catches what the foreman can't.

What it does

SiteSentinel is a pre-shift safety kiosk. A worker walks up. A Raspberry Pi camera runs YOLOv8 PPE detection on the device itself, no cloud needed. An ElevenLabs voice agent runs a three-question safety briefing in the worker's language. A local Gemma 4 model running on the kiosk MacBook fuses the PPE result with the briefing transcript into a fit-for-duty decision in under a second. Every decision is signed by our SafetyOfficerAgent, which has its own Auth0 identity separate from any user. Every event gets persisted to MongoDB Atlas as the compliance audit trail. The supervisor sees worker tiles update live on the dashboard.

Sponsor technologies in use

We built against four MLH sponsor categories, plus the main healthcare-track grand prize. Each sponsor's technology is load-bearing in the architecture, not bolted on for credit.

Best Use of Gemma 4 (Google). Gemma 4 (gemma4 e4b) runs locally on the kiosk MacBook through Ollama. It's the model behind every fit-for-duty decision: it takes the YOLO PPE result and the ElevenLabs voice transcript and emits a structured JSON verdict in around 700ms. We chose Gemma specifically because it's open weight and runs entirely on local hardware, which is the whole point. A cloud LLM in the safety path would be the wrong architecture for a job-site kiosk that has to keep working when connectivity dies.

Best Use of ElevenLabs. ElevenLabs Conversational AI runs the three-question pre-shift safety briefing through the official React SDK. Custom agent, Jarnathan voice, auto-end on the closing phrase. The transcript feeds directly into the local Gemma 4 decision. ElevenLabs is multilingual-ready out of the box, which matters for crews that don't all speak English.

Best Use of Auth0 AI Agents. Our SafetyOfficerAgent is a Machine-to-Machine application in Auth0 with its own client-credentials flow against a custom API audience. Every fit-for-duty decision fetches the agent's access token, verifies it against Auth0's JWKS, and returns the verified identity in the response. The dashboard renders the verified client ID and issuer on every decision card. The agent has its own scoped identity. It never holds a user key or master key. This is the demonstrable answer to "how should AI agents authenticate in safety-critical systems."

Best Use of MongoDB Atlas. MongoDB Atlas stores every PPE event from the Pi as the compliance audit trail. We use the official Node.js driver from the Next.js API. The dashboard read path is an in-memory ring buffer for resilience, but Atlas is the source of truth for the durable record. Compliance audit trails are exactly the kind of write-heavy, append-mostly workload Atlas is designed for.

Healthcare track (occupational safety). SiteSentinel is occupational health, framed as construction safety. OSHA reports show consistent year-over-year worker fatality rates in construction, with a meaningful share traceable to PPE non-compliance and unfit-for-duty incidents. We're targeting the prevention point rather than the post-incident reporting point.

How we built it

The Raspberry Pi 4B runs a Python loop using the Ultralytics YOLOv8n hard-hat detection checkpoint. Camera feed comes in over a Logitech Brio with OpenCV. Detected events are POSTed to the kiosk every 1.5 seconds.

The kiosk dashboard runs on a MacBook M4 Max with Next.js 16, TypeScript, and Tailwind. Three API routes do the heavy lifting: one ingests PPE events from the Pi and writes them to Mongo, one returns the ElevenLabs agent ID to the client, and one handles the fit-for-duty decision.

The voice briefing uses the official ElevenLabs React SDK with a custom agent we built for the three-question pre-shift script. The agent auto-ends after the closing phrase. We're using Gemini 2.5 Flash on the backend with the Jarnathan voice.

The fit-for-duty reasoning runs locally through Ollama with gemma4 e4b. The decision route fuses the PPE status and the voice transcript into a structured JSON verdict in around 700ms. No cloud LLM in the safety-critical path, ever.

For the agent identity layer, we set up SafetyOfficerAgent as a machine-to-machine application in Auth0, with its own client-credentials flow against a custom API audience. Every fit-for-duty call fetches the agent's access token, verifies it against Auth0's JWKS, and returns the verified identity in the response. The dashboard renders this as a lock badge so the supervisor can see exactly which identity signed each decision. The agent never holds a user or master key. It just has its own scoped identity.

For compliance, MongoDB Atlas stores every PPE event. The dashboard read path uses an in-memory ring buffer, so the live pipeline continues to work even if Atlas is briefly unreachable.

Challenges we ran into

Network reliability under demo conditions was the big one. We originally planned a wired Pi-to-MacBook ethernet path so the entire kiosk would survive a Wi-Fi drop. We got TCP working over USB-C Ethernet, but couldn't keep the Pi's source IP routing stable in the time we had. Pivoted to wifi only for the live demo and made the Mongo writes best effort so the dashboard keeps working when Atlas is unreachable.

Auth0 environment variables ate twenty minutes. We were getting a 401 from the token endpoint, and it turned out the new agent's credentials were in our env file under the wrong keys. Easy fix once we caught it, but a frustrating one to debug at 3 AM.

There's a race between the voice ending and the PPE state. The verdict fires the moment the voice transcript closes. If the PPE state ticks back to missing hat between briefing end and decision render, the dashboard shows a contradiction for a second. Known edge case, fixable, but not before submission.

Accomplishments that we're proud of

End to end edge to cloud pipeline working in 14 hours. Pi to Mac to Mongo to dashboard, plus the voice agent and the Auth0 verified local Gemma decisions.

The line "the agent never holds the master keys" is demonstrably true. Every decision card on the dashboard renders the verified Auth0 client ID and issuer right there on screen. It's not a slide. It's the actual response from a verified JWT.

The safety critical path runs on local hardware. The Pi YOLO detection and the Gemma 4 fit for duty decision both run without a cloud LLM in the loop.

What we learned

M2M client credentials with JWKS verification is the right primitive for AI agent identity in safety critical systems. We thought we'd need full Token Vault and step up CIBA for the prize, and we figured out that demonstrating a scoped agent identity is the actual win.

Building a hardware demo on conference wifi is a planning constraint, not a technical one. The architecture has to acknowledge that offline looks different in production (wired industrial networks) than it does at a hackathon.

Real product framing beats AI wrapper framing every time. We never described SiteSentinel as AI for construction. We described it as a kiosk that uses AI where AI is the right tool.

What's next for SiteSentinel

Production wired networking. Pi to kiosk over PoE ethernet, with the kiosk acting as the only DHCP server on a fully isolated industrial subnet. The wired path also fixes the wifi loss case we couldn't fully demo tonight.

Multilingual voice agent. ElevenLabs already supports it. We shipped English only for the hackathon. Spanish is the obvious next deployment given US construction crew demographics.

Vest detection. The current YOLO checkpoint is hard hat only. Adding hi vis vest detection is a one day model swap with a multi class checkpoint.

Auth0 step up CIBA. When the agent returns review or unfit, escalate to a human supervisor's phone for a step up confirmation before the worker gate physically opens. That's the next prize tier and the right production shape for a safety critical decision.

Mongo sync queue. Right now Mongo writes are best effort. A proper write ahead log on the kiosk would buffer events through wifi outages and replay them when the connection returns, guaranteeing a zero loss audit trail.

AI tooling disclosure

Per the hackathon rules: we used Claude (Anthropic) as a coding pair throughout the build, mostly for terminal level guidance, debugging, and architecture sounding board work. All architecture decisions, code reviews, demo design, and final integration were done by the team. The project description here is our own writing.

Built With

Share this project:

Updates