SafeGuard

A safety chip that sits between an untrusted AI and a robot's motors, and keeps the AI's mistake from turning into a human injury.

AI makes the device smart. QNX makes it dependable.

Inspiration

ISO 10218, the industrial robot safety standard, got rewritten in 2025. It was the first real update since 2011, and the reason is simple: robots are leaving their cages. Cobots, home robots, warehouse arms, all increasingly taking orders from a neural net, and all increasingly working right next to people.

Here's the part that stuck with us. You can't certify a neural net. There's no way to prove what it does on an input it has never seen, so you can't trust it to be the thing holding the stop button. Everyone is busy making robots smarter. Almost nobody is building the dumb layer underneath that makes them safe, the part that guarantees a glitchy or hijacked or straight-up-crashed AI sends the arm toward stillness instead of into someone's head.

So we built that layer, and we put it on QNX. When a person is about to get hit you have a few milliseconds and no time to phone a server. The safety has to live on the device, in real time. That's the actual reason QNX is in this project and not just Linux.

What it does

SafeGuard is a referee. It's a small, dumb, checkable piece of code that sits between an untrusted AI planner and the motor.

The AI asks for a motion. In our demo a real person plays the AI, because a real human is genuinely unpredictable and we didn't write them. Meanwhile SafeGuard watches the humans near the machine with its own on-device vision model. Then a deterministic arbiter, pure geometry and physics with no learning in it, decides whether that motion is safe and either lets it through, slows it down, or kills it.

Three things run at once, on purpose:

A push-away field. Every person the system sees is treated like a magnet the robot doesn't want to touch, so even leftover motion drifts away from people instead of toward them. It's always on, not a mode you flip.
Speed and separation. Far away, the robot runs full speed. Get closer and it creeps. Get too close and it holds. That's the core of ISO 10218:2025, written as a plain inequality.
A stopping-distance check. Before it commits, a physics model asks whether a full stop actually fits in the space that's left:

$$S = v_{human}\,t_{react} + v_{robot}\,t_{react} + d_{stop} + Z_{margin}$$

If the stop fits, it stops. If it doesn't, dodging becomes the main move and braking is the backup. It's reasoning about physics, not playing a scripted animation.

The demo shot we're proudest of: two lines on screen. One is the path the AI asked for. The other is the path SafeGuard actually allowed. The gap between them is the injury that didn't happen.

The one idea worth remembering

Most systems hand the smartest component the most control. We flipped that.

The more power a component has over a person's safety, the dumber and more checkable it should be.

  smartest  ->  least power   (the planner AI: it suggests, it decides nothing)
  dumbest   ->  most power    (the stop: verifiable, absolute, boring)

There are three players and we never let them blur together. The untrusted planner AI only suggests. Our on-device AI can only ever argue for less motion, never more. And our arbiter, the geometry-and-physics one, holds the actual veto and is deliberately not an AI. It earns the final say by being provable, not by being clever. This isn't a new idea in safety engineering, it's the Simplex / runtime-assurance pattern, the same approach Veo Robotics used for factory arms. We aimed it at the AI era.

How we built it

The safety brain is portable C99, written contracts-first. We froze the headers as the spec, then filled in ten modules behind them: geometry for separation, the push-away field, the stopping-distance gate, a PID loop with anti-windup, a jerk limiter so nothing moves violently, a fake motor to test against, and the supervisor itself. The supervisor is six rules in order, short enough to read start to finish in one sitting. That readability is the point. You can't call something a safety core if nobody can hold it in their head.

For testing we wrote 10 test files and 180 assertions, and they all pass. We also ran an adversarial review pass over the whole thing and it caught 13 real bugs, which was humbling. On top of that there's an integration harness that plays three scenarios (a person approaching, a sweep, and a hijacked command stream) and refuses to pass unless four things hold every single step: separation never drops below 0.20 m, the executed command stays in bounds even when we flood it with garbage, nothing ever goes NaN, and the arm goes to a safe hold the moment the vision feed goes stale.

Why QNX, specifically:

Scheduling is deterministic, so loop timing is tight, so the motion is smooth and the worst-case reaction time is something you can actually prove. We measure it as a jitter histogram while the CPU is under load.
The microkernel keeps processes in separate memory. The big crashy AI can't reach in and freeze the supervisor. Kill the AI and the guardian keeps running.
Message passing has priority inheritance, so the AI can't accidentally starve the control loop.
A dead process is a thing QNX notices and restarts, and the arm stays safe while that happens.

We wrote the real-time control loop for QNX SDP 8.0 on a Raspberry Pi (a pulse timer, FIFO scheduling, the jitter histogram) and lined up the on-device AI on the official TensorFlow Lite and OpenCV ports, the same stack QNX ships in its own camera sample.

There's also a browser demo. It runs pose detection live and drives a JavaScript copy of the exact same arbiter logic. The robot follows your hand, backs off from your head, and snaps to a hold the second you step out of frame. It's the whole idea, running on a webcam you can wave at.

Challenges we ran into

Keeping the AI out of the safety decision was harder than it sounds. Every few rules there'd be a spot where a model would obviously be "better," and every time we forced it back into a plain inequality. The stop is always physics, never a guess.

Making silence mean danger was the subtle one. A missing detection is the most dangerous input there is, because it's so easy to treat "I see nothing" as "all clear." We flipped it: the robot only runs fast when the vision is fresh and confident. Anything stale past half a second, or missing, or unsure, means slow down or stop.

We also had to make sure the safety move couldn't hurt anyone by itself. The dodge is rate and jerk limited too, so the thing meant to save you can't turn into a violent swing of its own.

And we built the entire brain without the hardware in hand. Keeping everything above a hardware layer, in plain C, with a simulated motor as the always-available fallback, meant the demo never depended on a servo showing up on time.

What we learned

Safety is an architecture problem before it's an algorithm problem. The win wasn't a clever model, it was deciding where the intelligence is allowed to live and keeping it away from the veto.

You can measure determinism. A jitter histogram turns "it's real-time, trust us" into a chart, and that's a much better argument.

And we came away with real respect for what a microkernel buys you. On a normal OS one wedged AI thread can drag everything down with it. On QNX the guardian just keeps going while the AI is dead. Seeing that live is what sold us on it.

What's next

Give our AI an actual brain: motion prediction so it can flinch before geometry alone would catch it, and anomaly detection so it can spot a hijacked command stream. AI watching AI.
Wire up a real joint, a servo and a status LED on the Pi's GPIO, so the two diverging paths move something physical instead of pixels.
Get the jitter histogram off the laptop and onto the Pi under load. That's our first real "measured on hardware" number.

A note on honesty: this isn't a safety-certified product, and we won't call it one. It's the certified industry safety architecture, built end to end and shown running. The arbiter is deterministic on purpose, the AI only ever votes for caution, and it never gets to authorize motion. That last part is the whole point.