Pathfinder

A real-time trace doctor for QNX 8.0: finds priority inversions, IPC deadlocks and starvation in kernel traces, explains the root cause and proves the fix, with every claim tied to trace events.

Comment

Inspiration

What it does

How we built it

Challenges we ran into

Accomplishments that we're proud of

What we learned

What's next for Pathfinder

Inspiration

In 1997 the Mars Pathfinder lander kept resetting itself. A low-priority task held a lock, a medium-priority task kept the CPU, the high-priority bus task starved and the watchdog fired. It took engineers days to find it in a trace. QNX ships an instrumented kernel that records every scheduling decision, but reading the trace by hand is slow. We wanted a tool that reads the trace for you.

What it does

Pathfinder reads a QNX 8.0 kernel trace (tracelogger / traceprinter) and tells you what is wrong, why, and how to fix it, with every claim tied to the kernel events that prove it.

  • Priority inversion: the full blocking chain (high-priority waiter, lock, owner, the unrelated thread that starved the owner), the duration, and whether priority inheritance was observed.
  • IPC deadlock: a wait-for cycle between servers (MsgSend/MsgReceive, matched by rcvid, coid and chid) and the clients stuck behind it.
  • Missed deadlines, CPU starvation and watchdog resets, linked into a causal chain: inversion, then deadline misses, then watchdog reset.
  • Fix proof: A/B traces. With PTHREAD_PRIO_INHERIT the report shows the lock owner boosted from priority 10 to 30 and no misses.
  • A single-file interactive HTML timeline (zoom, pan, click a finding), plus Markdown and JSON reports. An optional LLM narrative must cite event ids; claims with unknown ids are dropped.

It ships with four scenario programs for QNX 8.0 (Mars Pathfinder with a PTHREAD_PRIO_NONE mutex and with a semaphore, a two-server IPC deadlock, a long critical section, CPU starvation), each in bug and fix mode, plus ten real traces recorded on QNX 8.0.

How it uses QNX (evidence)

  • Runs on QNX SDP 8.0 (QNX Everywhere license), QEMU x86_64 quick-start image, instrumented kernel procnto-smp-instr; uname: QNX qnxqemu 8.0.0 2026/06/05 x86pc x86_64. Built with qcc 12.2.0 (-Vgcc_ntox86_64; sources also build for aarch64le).
  • QNX features used: instrumented kernel tracing and wide events, TraceEvent user events, SCHED_FIFO, PTHREAD_PRIO_NONE/INHERIT/PROTECT, _NTO_TCTL_RUNMASK, ChannelCreate, MsgSend, MsgReceive, MsgReply, MsgSendPulse, TimerTimeout.
  • scripts/e2e.sh rebuilds, boots QNX, runs all scenarios under tracelogger and asserts the findings (10 of 10 pass); 12 unit tests run on the committed real traces.

How we built it

C programs against the QNX 8.0 APIs, traces recorded on the real image, then a standard-library-only Python analyzer: parser, thread-state model, four detectors, explanation rules and an HTML renderer. The SDP only runs on x86 Linux/Windows, so on an Apple Silicon Mac we used an amd64 container for the toolchain and QEMU software emulation for the target (scripts included).

Build and run: QNX SDP 8.0, QEMU x86_64 quick-start image (no KVM needed), toolchain qcc 12.2.0. source ~/qnx800/qnxsdp-env.sh && make -C target, scripts/qnx-vm.sh start, scripts/pf-run.sh mars none 3, then python3 -m pathfinder analyze ... (full steps: docs/RUN.md in the repo). Without QNX, python3 -m unittest discover host/tests analyzes the committed real traces.

Challenges

  • QNX mutexes inherit priority by default, so reproducing Mars needs an explicit PTHREAD_PRIO_NONE or a semaphore; the tool says so instead of giving generic advice.
  • Uncontended locks never enter the kernel and semaphores have no owner, so ownership comes from kernel owner fields where possible and from lightweight markers otherwise.
  • TimerTimeout() arms only the next blocking call; a trace marker between it and MsgSend silently disabled our deadlock timeout.

Accomplishments

A real reproduction of the Pathfinder failure on QNX 8.0 with an automatic, evidence-linked diagnosis, an A/B fix proof, an IPC deadlock detector and a scripted end-to-end run on real traces.

What we learned

How much the instrumented kernel exposes: thread priorities on every event, mutex owners on contended locks and message ids that pair senders with receivers.

What's next

Resource-manager and pulse-aware wait-for graphs, aarch64 traces on Raspberry Pi, Momentics .kev import, CI use (exit status 1 on critical findings already works).

Codelab (bonus)

A 20-minute tutorial on catching a priority inversion with the instrumented kernel: https://github.com/seunghyeon1004/pathfinder-qnx/blob/master/docs/CODELAB.md

AI use disclosure

Developed with an AI coding assistant (Claude). All code was run and verified on QNX 8.0; the demo video uses synthetic narration and no music.

Built With

Share this project:

Updates

Submission history