SiteSense

Inspiration

Every year, preventable accidents injure workers in mines, on construction sites, and in warehouses — often because no one happened to be watching the right monitor at the right second. Safety supervision is expensive, exhausting, and impossible to do perfectly with human eyes alone. We asked a simple question: what if a single ordinary camera could watch everything, never blink, and raise the alarm the instant something went wrong — for miners, heavy-equipment operators, factory workers, and even ordinary citizens in public spaces? That idea became SiteSense.

What it does

SiteSense turns any ordinary webcam or CCTV feed into a complete AI safety monitoring system. A single video stream flows through a modular pipeline of 12+ hazard detectors — PPE compliance (GateGuard), person-down/fall detection, fire, zone intrusion, vehicle proximity, crowding, camera tampering, and more.

When a hazard appears, SiteSense:

Draws live annotations on the feed and locks/releases a virtual safety gate
Raises real-time alerts over WebSockets with severity levels
Saves a snapshot to an audit-ready evidence vault — with faces automatically blurred
Tracks analytics (compliance rates, event trends) and exports CSV/PDF reports

It runs entirely on local hardware — no cloud fees, no per-camera licenses, no specialized sensors — making enterprise-grade safety affordable for everyone from industrial sites to whole communities.

How we built it

Backend: Python + FastAPI, with a threaded video pipeline that runs YOLOv8 (Ultralytics) once per frame for person/vehicle detection, then fans each frame out to independent detector modules.
Computer vision: OpenCV for classical detectors (fire flicker analysis, tamper, zones) and MediaPipe's Pose Landmarker (Tasks API) for fall detection.
Real-time delivery: an MJPEG stream for the annotated video and a WebSocket event bus for instant alerts.
Frontend: React + Vite + Tailwind dashboard — live feed, event feed, module toggles, evidence vault, analytics charts, and an interactive danger-zone editor.
Storage: SQLite for events, with face-blurred snapshots written to disk. The whole thing is a plug-in architecture: a new hazard is just one new detector class.

Challenges we ran into

A shifting ML ecosystem: MediaPipe had dropped the legacy solutions API, crashing our fall detector. We migrated it to the newer Tasks API and built auto-download of the pose model.
False alarms everywhere: our fire detector lit up on skin tones and hi-vis vests, and fall detection fired on anyone sitting or bending. We re-engineered both — adding real temporal flicker analysis, person-region masking, and requiring both a horizontal pose and a wide bounding box for falls.
Event flooding: detectors fire every frame, drowning the feed in duplicates. We built an incident aggregator that collapses repeats into a single, count-tracked event.
A frozen backend: opening the webcam inside an async endpoint blocked the entire event loop. We moved device handling into the pipeline thread.

Accomplishments that we're proud of

A genuinely modular, plug-in safety platform — 12+ detectors on one shared pipeline and event bus.
Real-time everything: live video, instant alerts, and a dashboard where every panel updates live.
Privacy by design baked in from day one — faces blurred before storage, nothing ever leaves the machine.
Turning noisy, false-positive-prone prototypes into reliable, tunable detectors demo-ready on a laptop CPU.
Proving that high-coverage industrial safety can run on a single commodity camera at near-zero cost.

What we learned

Classical CV and modern deep learning are stronger together — YOLO for "who/what," pose and color/temporal analysis for "what's happening."
In safety systems, reducing false positives is as important as detection itself — an alarm that cries wolf gets ignored.
Real-time architecture is unforgiving: blocking calls, threading, and debouncing decisions make or break the experience.
Good UX (clean event feeds, dismissible alerts, live status) is what turns a model into a product.

What's next for SiteSense

Production-grade models: fine-tuned PPE and hazard detectors to replace heuristic fallbacks.
Multi-camera support with a unified, site-wide command dashboard.
Smarter incidents: detector fusion, severity escalation, and automatic notifications (SMS/Slack/siren).
Edge deployment on devices like Jetson for on-camera inference.
Broader reach: tailored detector packs for mining, construction, logistics, and community/public-safety use — because protecting people shouldn't depend on who's watching the monitor.