Inspiration

On QNX, processes talk by sending a message and waiting for the reply. The well-known rule for keeping that safe is the send hierarchy: sends go one way, and nobody sends back up. Nothing enforces it. One rarely used code path sends a message the wrong way, and two processes end up waiting on each other forever.

What bothered me is how quiet that failure is. Nothing crashes and nothing is logged. You are left running pidin, picking out the SEND- and REPLY-blocked threads, and following the chain by hand, and you still have to convince yourself it is really permanent and not just slow. I wanted a tool that does that reasoning for me, and that tells me about the bad design before it bites.

What it does

knotwatch watches a running QNX OS 8.0 system and reports five kinds of finding:

  • deadlock: a set of threads that no runnable thread can ever unblock
  • blocking-cycle: processes blocked on each other right now, not yet provably permanent
  • mutual-send: sends have been seen going both ways around a cycle
  • contract: a send or connection breaks the layering you declared in a small contract file
  • potential-cycle: connections form a cycle, though no message has crossed it yet

The last one is the point. In the demo, a brake controller and a wheel sensor run normally, and knotwatch already says the sensor holds a connection that goes up the hierarchy. A few seconds later the sensor hits its fault path, and the deadlock is reported about a second after it happens, with the two threads, what each is waiting on, and the statement that nothing can wake them.

It also follows mutexes and thread joins, so it catches the nastier case where a thread holds a lock while it sends a message: three threads, two processes, one deadlock.

Other things it does:

  • knotwatch watch is a QNX resource manager. Results appear as plain files under /dev/knotwatch (report, json, dot, snapshot), so anything can read them with cat.
  • knotwatch scan exits with status 0, 1 or 2, so a test script fails on a deadlock instead of hanging.
  • A snapshot taken on the target can be analysed again on any Linux or macOS machine, with no QNX tools.
  • A one-file viewer draws the send graph with servers above their clients.
  • It can give up root and keep only the two process-manager abilities a scan needs.
  • A small read-only server lets an AI coding assistant ask what is stuck and why.

How I built it

The core idea is that a plain cycle search is wrong for QNX. A client blocked on a server can be released by any thread of that server, so I compute which blocked threads still have a possible rescuer: a thread is alive if it is runnable, waiting for an outside event, or has a kernel timeout armed, and a blocked thread is alive if anything it waits on is alive. Whatever is left when that stops changing is a real knot.

The code is split in two on purpose. A QNX-only collector turns the system into a snapshot, and a portable core analyses snapshots. That let me unit-test the analysis on a normal machine with hand-written snapshots (also under ASan and UBSan) and keep the QNX-specific part small.

Everything is C, built with GNU make. The viewer is one HTML file with no dependencies, and the AI assistant server is one Python file using only the standard library.

How it uses QNX

  • QNX version: QNX OS 8.0.0, built with QNX SDP 8.0 (build 141)
  • Targets: x86_64 and aarch64le, both run in QEMU from images made with mkqnximage
  • Toolchain: ntox86_64-gcc and ntoaarch64-gcc from the SDP's Linux host tools, GNU make

Where each QNX feature is used:

QNX feature File and function
Thread states and block reasons, DCMD_PROC_TIDSTATUS on /proc/<pid>/ctl src/qnx/collect.c, collect_threads()
Connection enumeration with ConnectServerInfo() src/qnx/collect.c, collect_connections()
Mutex owner read from the waiter's address space, /proc/<pid>/as src/qnx/collect.c, resolve_lock_owners()
Resource manager with per-open pinned contents (extended OCB) src/qnx/watch.c, io_read(), ocb_calloc()
Timer pulse on the dispatch channel src/qnx/watch.c, kw_watch_run(), on_tick()
slog2 event logging src/qnx/watch.c, log_changes()
Dropping root while keeping two abilities, procmgr_ability() src/qnx/privs.c, kw_drop_privileges()
Native message passing, name_attach() / name_open() demo/brake_ctl.c, demo/wheel_sensor.c
A send under a kernel timeout, TimerTimeout() demo/wheel_sensor.c
A mutex held across MsgSend() demo/motor_ctl.c, demo/logger.c

Challenges I ran into

  • Building on a Mac. The SDP host tools are Linux programs, so I run them in a small container and boot the result in QEMU. The scripts for that are in the repo, so anyone on macOS can repeat it.
  • Finding out who holds a mutex. The process manager tells you a thread is waiting on a mutex, but not who owns it. I read the mutex out of the waiting process's memory and decode the owner from the value stored there. I worked that encoding out by observation and checked it against pidin on both architectures. It is not a documented interface, so when the owner cannot be determined for certain, knotwatch treats the thread as not stuck and says so.
  • False alarms from a stock system. The first runs flagged the process manager and the disk driver, which hold connections to each other by design. I excluded edges that start at the process manager, and left the other stock case detectable but silenced with one ignore line in the contract.
  • A clean-looking report from a system it could not see. Run without privileges, the first version read nothing and reported "no findings". It now refuses with a clear error instead.
  • **Not calling a

Built With

  • c
  • javascript
  • make
  • message-passing
  • python
  • qemu
  • qnx
  • qnx-os-8.0
  • qnx-sdp-8.0
  • resource-manager
  • slog2
Share this project:

Updates

Submission history