-
-
Overview: drive from the phone; lose the link and the robot stops on its own
-
Link loss → SAFE STOP, detected in 158.6 ms
-
E-STOP and HAM restarting the robot process
-
1000-trial stop-latency bench on QNX 8.0 (QEMU)
-
Real pidin and /dev/teleop output on the target
-
Architecture
-
Operator UI: connect, events, map
Inspiration
Remote-assist robots — warehouse AMRs waiting for a human to clear an obstacle, tele-rehab and medical robots, remote heavy equipment — all share one question: what happens when the link to the operator drops? On Wi‑Fi or cellular, a hiccup is normal. If the robot only stops when an app-level timeout fires on a busy general-purpose OS, the stop time depends on whatever else is running.
We wanted to show a stop whose worst-case latency is a design parameter, not an accident: a deadline on the heartbeat, evaluated by the highest-priority thread on a real-time OS. For context only (this is not a certification claim): industrial safety standards such as ISO 3691-4 (driverless industrial trucks) and ISO 13849 (safety-related control, incl. Safe Torque Off) expect a defined, time-bounded safe state. This project is a demo of that way of thinking, not a safety-rated product.
What it does
- An Expo / React Native app is the operator console: virtual joystick, top-down map with trail, a hold-to-arm control, a big E-STOP, a simulate link loss toggle, a status strip (mode, RTT, heartbeat age against the 150 ms deadline) and a latency panel fed by the robot's own statistics.
teleopd(C, QNX 8.0 aarch64le) is the robot-side daemon. The app sends a heartbeat at 20 Hz. If no heartbeat arrives for more than 150 ms the control loop entersSAFE_STOP: velocity is ramped to zero and latched until an explicitarmis accepted with a fresh heartbeat.estopstops immediately (no ramp) and always wins over anarmthat arrived earlier (receive timestamps). Losing the client while armed is also a safe-stop.simplantis a simulated differential-drive robot (200 Hz, 10×10 m) that talks toteleopdover QNX message passing./dev/teleopis a resource manager:cat /dev/teleopshows live status and detection percentiles;teleopctl set-deadline <ms>changes the deadline at run time throughdevctl.- Pairing: a 6-digit token and an
Origincheck. Scope is a trusted LAN demo (plainws://, no TLS), as documented indocs/ARCHITECTURE.md.
How we built it
- Two priorities, one rule. The net thread (WebSocket,
SCHED_RR10) never touches the control loop directly; it stores the message and sends a coalescedMsgSendPulse. The control thread (SCHED_FIFO50) blocks in oneMsgReceiveon a_NTO_CHF_FIXED_PRIORITYchannel and is woken by a 100 Hztimer_create+SIGEV_PULSEtick. The deadline is only checked on a tick, so the theoretical detection bound is 150 ms + 10 ms + jitter. - Own WebSocket (RFC 6455) in
qnx/common/ws.c: no third-party dependency on the target; non-blocking handshake with a 2 s deadline, send timeouts, RSV/oversize frames refused. - Message passing to the plant:
name_attach/name_open/MsgSendwithTimerTimeout(5 ms) so a hung plant cannot stall the safety loop; simplant has its own 200 ms watchdog. - Resource manager
/dev/teleopwithdevctlfor status/deadline/estop, andslog2logging. - Measure, don't claim:
tools/wsclient/wsclient.py --bench Nruns N heartbeat-loss trials and writes CSV;tools/fault/cpu-hog.sh+qnx/latbenchrun the priority experiments. - Development setup on Apple Silicon: QNX SDP 8.0 has no macOS host. We run the SDP in an
x86-64 Linux container (colima), build the image with
mkqnximage, and boot it in QEMU on the Mac (TCG, because HVF currently fails in QNX startup). Full recipe:docs/codelab/. - How AI was used: we built this with Claude (Anthropic) running as a multi-agent setup in
OpenCode. A planning agent drafted the architecture and schedule. Coding agents wrote most of
the C (teleopd, simplant, the WebSocket server, resmgr, HAM registration) and the React Native
app. Independent review and security agents then audited each milestone and found real bugs
that we fixed before publishing: an estop/arm pulse-ordering race, NaN timestamps bypassing
the stale-command check, a slowloris handshake stall, and the pairing token leaking into
slog2. A verification agent re-checked every milestone against its acceptance criteria. The
human side chose the idea after surveying existing entries, made the architecture and safety
calls (e.g. latch SAFE_STOP on
plant_lost), approved each milestone, and ran the physical-iPhone tests. Every number in this write-up comes from runs on the QEMU target or the iPhone, not from a model. Where measurement contradicted an AI prediction (a priority-50 CPU hog was expected to starve the watchdog on the 2-vCPU VM; it did not), we published the measured result.
Challenges
- No native QNX host on a Mac. SDP in an amd64 container, QEMU/TCG for the target. TCG emulates CPU and timers, so absolute jitter is not representative of real hardware.
- iOS refused to connect (403). React Native on iOS sends
Origin: http://<ip>:8080; our anti-drive-by Origin check rejected it. Fix: accept a same-host Origin only whenHostis an IPv4 literal orlocalhost, so DNS-rebinding pages are still refused (see CHANGELOG, Fixed). - Priority is everything. A watchdog is only as good as its priority relative to the rest.
We tested it on purpose: a hog at priority 10 does not disturb the loop (p99 160.82 ms over
1021 trials). Two hogs at the same priority (50) on both vCPUs hold the control thread
READYuntil they end: detection 3001–3045 ms with 3 s hogs, 5004–5040 ms with 5 s hogs (pidintraces indocs/BENCH.md). One hog, or hogs at 49, leave detection at 152–176 ms. The codelab sample first seemed to disagree, but its own heartbeat thread starved too, so it measured nothing during the hog. - Ordering races.
armandestoparrive on different pulses; we stamp receive times on the net thread soestopalways wins. - HAM. The HAM package was not installed in our SDP at first; supervision/restart status
after install: HAM restarts simplant, slay →
plant_ok=0in 38–88 ms,plant_ok=1again in min 544 / median 571.5 / max 626 ms (n=10, QEMU TCG); while the plant is down an armed session getsplant_lost→ SAFE_STOP, latched until a newarm.
Accomplishments we're proud of
| Result | Value | Source |
|---|---|---|
| Heartbeat-loss detection, QEMU TCG, 20 and 25 trials | median 154.8 / 156.5 ms, max 159.4 / 160.4 ms (a later run peaked at 161.0 ms) | docs/BENCH.md |
Detection with deadline changed to 200 ms via devctl |
p50 202.9 ms, max 210.3 ms (20 trials) | docs/QNX_EVIDENCE.md |
| Baseline, 1000 trials (QEMU TCG) | p50 155.11 / p99 161.01 / max 174.05 ms (2 trials > 170 ms) | docs/BENCH.md §Safe-stop detection latency |
CPU hog, 2 × SCHED_RR 10, 1021 trials |
p50 155.11 / p99 160.82 / max 232.55 ms (3 trials > 170 ms) | docs/BENCH.md §Safe-stop detection latency |
CPU hog, 2 × SCHED_FIFO 10 (1 trial) |
158.2 ms | docs/BENCH.md §Priority experiments |
CPU hog, 2 × SCHED_FIFO 50 in teleopd (misconfiguration; 6 trials 3 s, 3 trials 5 s) |
3001–3045 ms / 5004–5040 ms (= hog duration); 1 hog: 152–158 ms | docs/BENCH.md §Priority-50 investigation |
Physical iPhone: hb_timeout |
156.02 ms | docs/BENCH.md §Physical device test |
| Physical iPhone: E-STOP handled; RTT | 0.07–0.18 ms; 7–42 ms | docs/BENCH.md §Physical device test |
Plant killed → plant_ok=0; HAM restart → plant_ok=1 (n=10) |
38–88 ms; min 544 / median 571.5 / max 626 ms; plant_lost → SAFE_STOP latched |
docs/BENCH.md §Plant failure |
Also: a standalone, compilable deadline-watchdog codelab (docs/codelab/) so others can
reproduce the pattern on QNX 8.0 under QEMU on Apple Silicon.
What we learned
- A real-time loop is defined by who can preempt it. A hog at priority 10 leaves detection at
p99 160.82 ms; two hogs at priority 50 delay it by exactly as long as they run (3 s → ~3.0 s,
5 s → ~5.0 s), because
SCHED_FIFOnever time-slices between equal priorities. - Check what a benchmark can see: our codelab sample "passed" with priority-50 hogs only because its heartbeat thread starved too, so it never measured a trial during the hog.
_NTO_CHF_FIXED_PRIORITYon the control channel keeps the receiver at priority 50 even when a priority-10 thread sends it a pulse.- Pulses are a good fit for "tell the real-time thread something happened without ever blocking the sender"; coalescing with pending bits keeps the pulse rate bounded.
- Different failures deserve different stops:
estopis deliberate (stop now), a link loss is ambiguous (bounded ramp, to avoid shifting a load). - Emulation tells you the logic is right and the bound is respected; it does not replace measurements on real hardware.
What's next
- Guard against priority misconfiguration: refuse to arm (or log) when another CPU-bound thread
runs at ≥ the control priority, and trace the pulse queue during such a stall (
pidinshowed controlREADY; the queue was not traced). - Run on real QNX aarch64 hardware (QEMU HVF is blocked today by a GICv3 assert in startup).
- TLS and stronger pairing for anything beyond a trusted LAN.
- Real plant (motor controller / AMR) behind the same message interface.
Build instructions (required by Devpost)
- QNX version / target: QNX SDP 8.0,
aarch64le, QEMUvirtmachine (cortex-a57, 2 vCPU, TCG). - Toolchain:
qccfrom SDP 8.0 (gcc_ntoaarch64le), run in an amd64 Linux container. - Host: macOS arm64, Homebrew
colima docker qemu; free myQNX account and QNX Everywhere non-commercial license (seedocs/codelab/README.md, §2–4).
tools/sdp/run.sh /work/tools/sdp/make-image.sh # QNX 8.0 image
tools/sdp/run.sh make -C qnx # teleopd, simplant, teleopctl, latbench
QEMU_SERIAL=build/qnx-image/serial.log tools/qemu/run-qemu.sh &
TELEOP_TOKEN=123456 tools/qemu/deploy.sh
python3 tools/wsclient/wsclient.py --token 123456
cd app && npm install && npx expo start
QNX evidence
Locations checked with grep against the current source (line numbers drift as files change;
use the function name). docs/QNX_EVIDENCE.md is the master list.
| File:function | QNX API | Purpose |
|---|---|---|
qnx/teleopd/teleopd.c:main (ChannelCreate L1230) |
ChannelCreate(_NTO_CHF_FIXED_PRIORITY) |
Control channel keeps the prio-50 receiver at its priority |
qnx/teleopd/teleopd.c:set_sched (L220; used L496, L1096, L1262) |
pthread_setschedparam, pthread_setname_np; FIFO 50 / RR 10 |
Threads control 50f, net 10r, resmgr 10r |
qnx/teleopd/teleopd.c:control_main (L503–504) |
SIGEV_PULSE_INIT, timer_create(CLOCK_MONOTONIC) |
100 Hz tick as a pulse |
qnx/teleopd/teleopd.c:control_main (L511) |
MsgReceive |
Single wait point for tick/hb/cmd/arm/estop |
qnx/teleopd/teleopd.c:pulse_once (L575–586) |
MsgSendPulse |
Net → control, non-blocking; estop at prio 50 |
qnx/teleopd/teleopd.c:now_ns, on_tick (L161, L376) |
clock_gettime(CLOCK_MONOTONIC) |
Heartbeat age vs deadline; stats_add (L242) / stats_get (L267) |
qnx/teleopd/teleopd.c:plant_exchange (L336–353) |
name_open, TimerTimeout, MsgSend, name_close |
Velocity ↔ pose with a 5 ms bound |
qnx/teleopd/teleopd.c:resmgr_main (L1089–1135) |
dispatch_create_channel, iofunc_attr_init, resmgr_attach("/dev/teleop") |
Resource manager in its own thread |
qnx/teleopctl/teleopctl.c:do_devctl (L43–58) |
devctl() |
status, set-deadline, estop |
qnx/simplant/simplant.c:main (L109) |
name_attach |
Plant server (name_attach("simplant")) |
qnx/simplant/simplant.c:main (L106, L119–120, L131) |
pthread_setschedparam(SCHED_FIFO), SIGEV_PULSE_INIT, timer_create, MsgReceive |
200 Hz tick and request loop |
qnx/latbench/latbench.c:hog (L41, L47) |
pthread_setschedparam |
CPU hog for the priority experiments |
qnx/teleopd/teleopd.c:log_init/logf_ (L191–201, L143–150); qnx/simplant/simplant.c (L68–78, L47) |
slog2_register, vslog2f |
Logs via slog2info |
Runtime proof (QEMU TCG, 2026-11-14, from QNX_EVIDENCE.md): pidin -p teleopd → net 10r,
control 50f, resmgr 10r; ls -l /dev/teleop → crw-r----- 1 root root.
HAM (qnx/hamreg/hamreg.c L76–89: ham_attach, CONDDEATH, ham_action_restart): slay → plant_ok=0 38–88 ms; plant_ok=1 again min 544 / median 571.5 / max 626 ms (n=10).
Built With
- c
- expo.io
- qemu
- qnx
- react-native
- typescript
- websocket
