SentinelX — Autonomous Workplace Safety Agent

• 

  dashboard.png

• 

• 

• 

• 

Inspiration

Every year, thousands of workplace incidents are preventable — not by better cameras, but by better decisions. The current generation of safety CV systems operates like a smoke alarm with no snooze button: frame-wise detection, fixed thresholds, and a flood of alerts that train operators to ignore them. Within a week, these systems get switched off.

We built SentinelX because the hard problem in workplace-safety computer vision is not detection accuracy. It is alert fatigue. A system that cries wolf becomes worse than no system at all. We asked: What if the supervisor could remember, reason, and calm down — just like a human safety officer?

What It Does

SentinelX is a temporal risk agent, not a frame-wise detector. Every person on the floor receives a persistent identity. Violations accrue against that identity over time through a decaying risk integrator, and the agent only acts — speaks, logs, escalates — on a change in risk trajectory, never on a single noisy frame.

The Escalation Ladder

• Observe (< 0.25) — Silent monitoring.
• Nudge (≥ 0.25) — Spoken reminder.
• Warn (≥ 0.50) — Spoken warning + structured log entry.
• Escalate (≥ 0.80) — Incident card generated with cropped evidence snapshot and full JSON risk timeline.

The De-escalation Path (the detail that proves this is an agent, not an alarm) When a worker corrects their violation, risk decays exponentially. Once it clears the hysteresis band (0.12 below the rung floor) and holds for 1.5 s, the agent autonomously stands down, marks the incident resolved, and returns to silence — entirely without human intervention.

How We Built It

Architecture

The pipeline runs in five layers:

1. FrameSource — Camera, file, or synthetic scene generator (zero-hardware demo guarantee).
2. Perception — YOLOv8n detector (or deterministic stub), HSV-based PPE classifier with EMA smoothing, and MediaPipe pose estimator (or aspect-ratio fallback).
3. Tracking — Greedy IoU assignment with minHits/maxMissed gating, producing stable PersonTrack identities.
4. Cognition — ZoneEngine (ray-cast point-in-polygon), RiskModel (exponential decay integrator), and Escalator (4-rung ladder with hysteresis).
5. Action — VoiceSink (Web Speech API), IncidentStore (evidence-grade ledger), DashboardSink, and IoT corroboration.

The Temporal Risk Model

The core contribution is a continuous risk accumulator:

$$ \text{deficit}(c) = \max(0, 1 - c - \text{deadband}) $$

$$ \text{evidence}(t) = w_{\text{helmet}} \cdot \text{deficit}{\text{helmet}} + w{\text{vest}} \cdot \text{deficit}{\text{vest}} + w{\text{zone}} \cdot \mathbb{1}{\text{restricted}} + w{\text{prone}} \cdot \mathbb{1}{\text{prone}} + w{\text{iot}} \cdot \text{iot_hazard} $$

$$ \text{risk}(t) = \text{clamp}\left( \text{risk}(t-1) \cdot e^{-\Delta t / \tau} + \text{evidence}(t) \cdot \frac{\Delta t}{T_{\text{rise}}},\ 0,\ 1 \right) $$

Capability Probe — Degrade, Never Crash

Every optional backend is wrapped in a runtime probe. Missing dependencies result in available: false and a deterministic stub takes over. The pipeline completes regardless of whether the camera, model weights, speech API, or IoT bridge are present.

Challenges We Faced

1. The Latency vs. Stability Trade-off
   Adding temporal integration introduces ~200 ms of decision latency. We tuned $\tau$ and $T_{\text{rise}}$ through live benchmark simulation (43,200 frames = 30-minute shift) to ensure we did not sacrifice responsiveness for stability.

2. Zero-Dependency Demo Guarantee
   Hackathon demos fail when judges wait for model downloads. We built a pure Canvas 2D synthetic floor scene (52 s scripted loop) that renders compliant and non-compliant workers, restricted zones, and falls — all without a camera, GPU, or network call.

3. HSV PPE Classification in Uncontrolled Lighting
   Without bundling a fine-tuned classifier (to keep the demo weightless), we used HSV thresholding with exponential moving average smoothing. It is intentionally lighting-sensitive in the demo, but the architecture leaves a clean hook for a production classifier head.

4. Identity Persistence Through Occlusion
   Pure IoU tracking splits tracks on long occlusions, resetting accumulated risk. We mitigated this with conservative gating parameters, and documented the fix path (appearance embeddings) in our roadmap.

What We Learned

• Alert fatigue is a systems problem, not an accuracy problem. Reducing false positives by 10% does not help if the operator still receives 200 alerts per shift. Temporal integration reduced delivered alerts by an order of magnitude in our benchmark.
• Autonomous de-escalation builds trust. A safety system that can calm itself down is more likely to stay online than one that only knows how to scream louder.
• Deterministic stubs are a design pattern. Building every module with a guaranteed fallback turns "demo risk" into "demo certainty."

What's Next for SentinelX

• Per-zone risk parameters: A press bay deserves a faster rise time than a walkway.
• Appearance embeddings: Replace IoU-only tracking to survive long occlusions without identity fragmentation.
• Multi-shift dataset validation: Replicate the alert-reduction benchmark on real factory-floor footage across full 8-hour shifts.
• Edge deployment: Port the browser reference architecture to the Python/Flask/SQLite edge stack documented in our repo.

Built With

• affiliation
• agpl-3.0)
• build
• challenge-hackathon
• club-community
• computer
• corroboration
• domain
• estimation
• fallback
• hack
• iot-sensor
• layer
• mediapipe-pose
• prism
• react-ui-framework-typescript-type-safe-agent-architecture-vite-build-tooling-tailwind-css-voltagent-derived-dark-design-system-canvas-api-synthetic-vision-stage-rendering-web-speech-api-non-blocking-voice-actuator-yolov8-perception-backbone-(optional
• vision-core