Worker Safety Agent

• 

  stereo cam

• 

• 

  drashboard

Inspiration

We started by building AI-powered rehab monitoring technology — tracking a patient's movement to help physical therapists measure recovery. But while pursuing a pilot with a rehab clinic, we ran into something every early medtech team eventually hits: clinical deployment means real regulatory weight, and that's a long road for a small team.

Around the same time, we had a conversation with a small injection molding manufacturer connected through our incubation center. Their problem was strikingly similar to the one we'd already solved the hard part of: workers doing repetitive, physically demanding manual tasks in a cramped space, with a supervisor who can only watch one thing at a time. The same underlying technology — track a human's joints accurately, understand what the angles mean — pointed at a second, faster-moving opportunity: factory floor ergonomic safety.

We pivoted, and built something we think is genuinely different from a typical monitoring dashboard: not just a system that tells a human something's wrong, but an AI agent that decides what to do about it.

What it does

The system tracks a worker's joint angles via computer vision, scores ergonomic risk using RULA (Rapid Upper Limb Assessment) — a real, published occupational health standard from 1993, not an invented threshold — and feeds that risk data to an autonomous Gemini-powered agent. The agent doesn't just classify the data; it decides, live, whether to log the observation, notify a supervisor, or escalate immediately, and executes that decision with a full timestamped audit trail.

We deliberately gave the agent a hard authority boundary: it can inform, log, and escalate to a human — and nothing else. It has no tool that can stop machinery, discipline a worker, or take any action that directly affects a person. That boundary is enforced by which tools exist for it to call, not just a prompt instruction telling it to behave.

How we built it

• Perception: stereo/mono camera tracking of body joint positions, converted to real angles via vector geometry.
• Assessment: a deterministic RULA scorer — safety-critical logic stays fully rule-based and explainable, never left to an LLM's judgment alone.
• Decision layer: a Gemini agent with exactly four defined tools (log_incident, notify_supervisor, escalate_critical, generate_shift_report), plus a deterministic fallback that force- triggers escalation for any critical-risk worker even if the LLM call fails or the model doesn't follow instructions — we never wanted a safety-critical action depending on the AI alone getting it right.
• Visualization: a skeleton overlay that's color-coded by the live risk assessment, so perception and decision-making show up as one unified picture, not two separate systems a viewer has to connect mentally.
• Deployed via Google Cloud Run, with a Colab notebook for live camera demos.

Challenges we ran into

We tried to validate everything against known-correct scenarios before trusting it, and that discipline caught real bugs we're glad we found before they shipped:

• A shoulder-angle calculation that was silently inverted — a neutral, relaxed posture was scoring as maximally flexed. Only caught by testing against a synthetic pose with a known correct answer.
• A skeleton-label collision bug where labels overlapped when workers stood close together — our first fix (a distance threshold) looked reasonable and was still wrong; the real fix needed actual bounding- box collision detection.
• An initial-state bug in our continuous shift monitor that crashed on the very first reading, because we'd assumed a starting risk level that was never actually observed.

None of these were exotic failures — they were the ordinary kind of bug that only surfaces when you actually run the code against a scenario with a known right answer, rather than trusting that it looks correct.

What we learned

Grounding safety-critical scoring in a real, published standard (RULA) instead of an invented one gives the system actual credibility with anyone who knows the domain. And an agent that knows the limits of its own authority — escalating to a human rather than trying to handle everything itself — is a more trustworthy design than one with unlimited reach, even though it's less flashy to demo.

What's next

We're in an early, real pilot with a manufacturing partner connected through our incubation center, with a clear path to a recurring engagement contingent on the pilot's results. Next up: multi-worker floor-wide monitoring, a learned precursor model layered on top of (not replacing) the deterministic safety core, and validating the system against a broader range of real factory conditions.

Built With

• api
• raspberry-pi