qshim

Fault injection, record and replay in front of an unmodified QNX 8 driver. No rebuild of the app or the driver.

Comment

Inspiration

Code that talks to a device has error paths: a short read, an EAGAIN, a read that takes a second too long. Real devices rarely fail on demand, so those paths are rarely run before they matter. On QNX a driver is a resource manager, an ordinary process that owns a pathname, and the pathname space lets a second process attach in front of the first. That is enough to build a test tool that needs no change to the application and no change to the driver.

What it does

qshim attaches to the pathname of a running resource manager (for example /dev/fakesensor) with _RESMGR_FLAG_BEFORE. New open() calls of unmodified applications resolve to qshim, and qshim forwards every request to the real driver, which keeps running.

  • Fault injection. A scenario file is a list of one-line rules, such as on read nth=3 do short 6 or on read nth=4 do delay 1500. Actions: an errno, a delay, a short read, end of file, corrupted or replaced data. Rules can match by operation, devctl command, client pid, and count, and can be seeded for repeatable random faults.
  • Record. Every request and its answer is written to a JSON Lines trace.
  • Replay. qshim -R trace.jsonl serves the pathname from the trace alone, with the driver stopped.
  • Control device. /dev/qshim/<name> reports counters and accepts pause, resume, clear, and new rules at run time.
  • Host tools (Python, standard library only): a scenario lint and a trace report that matches each injected fault to the scenario line that caused it.

The demo is a small thermostat client with three bugs. Three one-line scenarios expose a false frost alarm from a short read, an early exit on a single EAGAIN, and a control loop stalled by one slow read. A fixed version survives all three. The same session is then replayed with the driver stopped.

How we built it

  • Target: QNX SDP 8.0, QNX 8.0.0 x86_64 under QEMU. Toolchain: qcc -Vgcc_ntox86_64. Host: WSL2 Ubuntu. The build and test steps are in the README.
  • About 2,000 lines of C for the target, plus the Python host tools.

Where QNX is used (files in the repository):

  • src/main.c: dispatch_create(), resmgr_attach() with _RESMGR_FLAG_BEFORE, and a thread_pool that serves requests.
  • src/backend.c: for each client open, ConnectAttach_r() to the real driver's channel and an _IO_CONNECT message sent with MsgSend_r(), so that qshim holds its own connection to the driver per open.
  • src/passthru.c: iofunc handlers that forward read, write, devctl and stat with resmgr_msgread() and MsgSendv_r(); devctl payloads are forwarded without interpreting them.
  • src/fault.c and src/passthru.c: unblock handling. A client that is interrupted while qshim is delaying its request is released, and an unblock during a driver call is passed on to the driver.
  • src/control.c: a second resource manager for the control device.
  • policy/qshim.example.txt and test/secpol/: a security policy fragment. Under a process type with no allow_attach rule for the path, the kernel refuses the attach and qshim exits with status 2 and names the missing rule.

Challenges we ran into

  • A proxy that only forwards messages on one shared connection breaks per-open state in the driver. qshim opens the driver once per client open by sending the connect message itself.
  • A delay injected by qshim must not make the client unkillable, so the delay has to cooperate with the kernel's unblock pulse.
  • A refusal by the security policy arrives as EACCES, while a missing pathspace ability arrives as EPERM. qshim reports both with a hint.

Accomplishments that we're proud of

  • The application and the driver in the demo are the same binaries with and without qshim.
  • Checked on the target under emulation: 123 target checks, 20 security policy checks, 17 demo checks, and 88 host unit tests.
  • A seven-step codelab (codelab/index.md in the repository) that builds a transparent _RESMGR_FLAG_BEFORE proxy from an empty file, with the code of every step and a script that builds and runs each step on the target (32 checks).

What we learned

The resource manager model makes interposition a matter of pathname resolution order. No kernel module and no preload library is involved; the tool is one more process, and the security policy decides whether it may attach.

What's next

  • Larger payloads in replay (reads above 256 bytes are truncated in the trace today).
  • Tests on hardware and on aarch64. The target ran under emulation (QEMU without KVM), so the times shown in the video are not latency measurements.
  • Clients that run as non-root or under restricted process types have not been tested yet.

Built With

Share this project:

Updates

Submission history