Inspiration

I have spent 18 years keeping mobile networks running in Rwanda. One failure keeps coming back: the grid drops at a remote tower, the diesel generator does not start, and the battery quietly runs flat. The site goes off air, often at night and hours from the nearest technician. The remedy is simple: notice within milliseconds, start the generator, and if it fails, shed the cooling load so the battery lasts for the radios. But it has to happen every time, on time, without a person watching. That is a job for a real-time operating system.

What it does

towerwatch samples a −48 V telecom power plant (mains, generator, rectifiers, battery, loads, cabinet temperature) 1,000 times a second on QNX 8.0 and protects the site:

  • declares a mains failure after 5 consecutive low samples (4 ms after the fault at 1 kHz), so a one-sample blip does not trip it;
  • requests the generator after a 3 s ride-through and raises a generator failure if it is not online within 20 s;
  • sheds the non-critical cooling load on battery when the generator has failed, the charge is below 50 % or the bus is below 47.5 V, retries the generator when it is repaired and restores everything after the bus has been healthy for 10 s;
  • raises and clears alarms with hysteresis and writes every event to the QNX system log;
  • measures its own wake-up latency, execution time and every missed deadline on every tick, so the timing can be checked on the target.

Recorded on QNX 8.0, same outage twice (mains lost at 10 s, generator fails until repaired at 60 s, mains back at 100 s): with towerwatch in control the mains failure was declared 4.002 ms after the fault and the site stayed up the whole time; with monitoring alone the battery ran flat and the site was off air for 31.6 s.

How we built it

  • 1 kHz loop: timer_create(CLOCK_MONOTONIC) with SIGEV_PULSE_INIT delivers a pulse to a private channel (ChannelCreate(_NTO_CHF_PRIVATE)); a SCHED_FIFO priority-40 thread with mlockall() waits in MsgReceivePulse. Tick work is timed with ClockCycles().
  • The plant as a driver-style process: pwrsim is a resource manager at /dev/pwrsim (resmgr_attach, io_read, io_devctl). towerwatch -R reads it over QNX message passing and sends generator and load-shed commands with devctl(); pwrctl injects faults live.
  • Logging: slog2_register and slog2f; slog2info shows the alarms in the system log.
  • Portable core: the protection logic and the plant model are plain C with no OS calls, so the same code runs in the QNX real-time loop, a Linux host build and the unit tests.
  • Browser-only route: tools/qnx-cloudshell.sh follows QNX's documented unattended install of QNX SDP 8.0 in Google Cloud Shell, builds with qcc, makes the image with mkqnximage, boots it in QEMU and runs the demo over ssh.

Where QNX 8.0 is used (for reviewers)

  • src/towerwatch.c, sampler(): timer pulse loop (timer_create, SIGEV_PULSE_INIT, ChannelCreate, MsgReceivePulse); work_clock(): ClockCycles() and SYSPAGE_ENTRY(qtime); log_event(): slog2f; main(): SCHED_FIFO, mlockall, slog2_register.
  • src/pwrsim.c: resource manager at /dev/pwrsim (resmgr_attach, iofunc_func_init, io_read with MsgReply, io_devctl), plant stepped by its own timer pulse.
  • src/pwrctl.c, include/pwrsim.h: devctl() commands (__DIOT DCMDs) for fault injection and status.
  • Makefile (make qnx, qcc -Vgcc_ntox86_64 or gcc_ntoaarch64le), image/towerwatch.build (buildfile lines), tools/qnx-cloudshell.sh (SDP install, mkqnximage image, QEMU boot, demo runs).
  • docs/qnx/series_qemu.txt and docs/qnx/outage_qnx.png: the samples and chart recorded on the QNX 8.0 target.

Build and run

QNX version: QNX SDP 8.0 (QNX OS 8.0.0). Target: QEMU x86_64 with the image from mkqnximage --type=qemu (a Raspberry Pi 5 build is make qnx QARCH=gcc_ntoaarch64le). Toolchain: qcc from QNX SDP 8.0. Steps are in the README; the browser-only route is bash tools/qnx-cloudshell.sh install then bash tools/qnx-cloudshell.sh all.

Challenges we ran into

  • Google Cloud Shell has no KVM and no network bridge, and mkqnximage --run needs a bridge. I start the same QEMU command that mkqnximage's runimage builds, with TCG emulation and user-mode networking.
  • CPU model: QNX 8.0 did not start on QEMU's Nehalem model, and with -cpu max its sshd crashed under emulation; IvyBridge boots in about 10 s with ssh working.
  • QNX Software Center refuses a symlinked ~/.qnx; a bind mount from the larger disk solved it.
  • The monitor first read the resource manager with pread(), which my io_read refused. It now uses read() and the resource manager accepts both.

Accomplishments that we're proud of

  • It builds with qcc and runs on QNX 8.0: the site stayed up through an outage that kept it off air for 31.6 s without control.
  • Two-process version over message passing, with live fault injection and alarms in slog2.
  • Anyone with a browser and a free QNX Everywhere licence can rebuild and rerun it end to end.

What we learned

Pulses and private channels make a periodic loop that is easy to reason about, and a resource manager turns a simulated plant into something the controller talks to exactly as it would a real driver. Timing measured under emulation must be labelled as such: in the QEMU run the wake-up latency was 244 µs mean and 787 µs p99 with 4,594 of 110,000 periods skipped, which measures the emulator, not QNX; the plant-as-resource-manager run declared the fault 21.3 ms after it happened (two processes, two 1 kHz timers, emulated CPU).

What's next

  • Run on a Raspberry Pi 5 with QNX 8.0 for hardware timing figures.
  • Replace pwrsim with a driver for a real site controller (Modbus over RS-485) behind the same /dev interface.
  • Report energy and diesel use per site from the counters towerwatch already keeps.

Built With

  • c
  • google-cloud-shell
  • matplotlib
  • mkqnximage
  • posix-timers
  • python
  • qcc
  • qemu
  • qnx
  • qnx-sdp-8.0
  • resource-manager
  • slog2
Share this project:

Updates

Submission history