page-fault-interceptor

A C++20 library and benchmark suite for userspace page fault handling: **what does it cost, and when is it the only option?**

Comment

Userspace page fault interception on Linux userfaultfd

A C++20 library and benchmark suite for userspace page fault handling: what does it cost, and when is it the only option?

Benchmark results

What this is for

userfaultfd lets a process intercept its own page faults and decide how to resolve them from ordinary userspace worker threads. This is fundamentally a control and capability mechanism, not a mechanism to make ordinary anonymous paging faster.

Where userspace fault handling is the only viable option

  1. Compressed memory tiers (CompressedBacking):
    • The region's pages live compressed in a userspace store. A page is materialised only when touched, and evicted back to the compressed store when the resident set exceeds a budget.
    • Why the kernel can't do this: The page contents do not exist anywhere the kernel can fetch them — they must be reconstructed by running arbitrary userspace decompression algorithms.
    • Why mprotect + SIGSEGV can't do this safely: Decompressing requires scratch buffers and taking mutexes on the store. Inside a SIGSEGV signal handler, taking locks can deadlock against the interrupted thread, and dynamic memory allocation is not async-signal-safe. With userfaultfd, faults are served on normal worker threads with full support for locking, buffering, and I/O.
  2. Copy-on-write live snapshotting (CowSnapshot):
    • Converting a single massive stop-the-world stall into tiny distributed fault delays during background snapshot materialisation, reducing peak writer stall from 6.9 ms down to 340 µs.
  3. Remote & lazy paging systems:
    • Post-copy VM live migration (QEMU), CRIU lazy process restore, Firecracker microVM snapshot restore, and far-memory tiers where pages are served across networks or remote memory.

Capabilities & Trade-Offs

1. Compressed Memory Tier: RSS vs. Latency Trade-Off

Rather than measuring nanoseconds vs. nanoseconds, the compressed tier trades physical memory footprint (RSS) against access latency. Honest accounting tracks compressed blobs, fragmentation, index overhead, and dirty writeback activity.

Arm Memory Held (RSS) vs Region Compression Ratio Hit Rate p50 Latency p90 Latency p99 Latency
Plain anonymous (control) 128.8 MB 100.6% 1.00x 100.0% 123 ns 290 ns 681 ns
LZSS (no eviction / 100% budget) 113.5 MB 88.7% 2.46x 84.7% 213 ns 8.0 µs 16.9 µs
LZSS (50% resident budget) 112.5 MB 87.9% 2.46x 84.7% 219 ns 7.7 µs 15.0 µs
LZSS (25% resident budget) 89.7 MB 70.1% 2.45x 74.6% 262 ns 11.8 µs 28.7 µs
zlib-1 (25% resident budget) 63.2 MB 49.4% 3.28x 74.6% 298 ns 21.2 µs 89.5 µs

Compression & Eviction Tracking

The tier maintains detailed telemetry exposed via CompressedBackingStats:

  • Compression Amount & Ratios:
    • LZSS block codec: ~2.46x ratio on structured record memory, optimized for fast decompression (~8–15 µs fault resolution).
    • zlib (level 1): ~3.28x ratio, achieving ~50% total physical memory reduction at the cost of higher decompression compute (~21–90 µs fault resolution).
  • Physical Memory & Resident Set Enforcement:
    • resident_pages & peak_resident_pages: Strictly bounds materialised frames to resident_budget_bytes.
    • stored_bytes: Actual compressed payload bytes stored across the chunked arena.
    • wasted_bytes: Bytes stranded by page rewrites that outgrew existing slots (fragmentation telemetry).
    • metadata_bytes: Per-page slot index footprint (page_count * sizeof(Slot)).
    • total_overhead_bytes: Complete tier cost (stored_bytes + wasted_bytes + metadata_bytes).
  • Dirty Tracking & Eviction Telemetry:
    • Missing pages are installed write-protected on reads (UFFDIO_COPY_MODE_WP).
    • Clean Evictions: Unmodified read-only pages skip recompression during eviction (madvise(MADV_DONTNEED) only).
    • Dirty Writebacks: Pages written to trigger write-protect faults (UFFD_PAGEFAULT_FLAG_WP), marking them dirty so they are re-compressed into the arena on eviction.

Architecture

Five decoupled layers, each usable and testable independently:

handlers/          ZeroFill · StagedCopy · CowSnapshot · CompressedBacking
                   what to install or reconstruct for a given fault
        |
   FaultHandler    the strategy interface (uffd/handler.hpp)
        |
     Engine        the event loop: workers, batching, wait strategy,
                   metrics, and the wake-on-failure guarantee
        |
     Region        an mmap + a uffd + the registration binding them (RAII)
        |
    sys / Mapping  1:1 syscall wrappers returning Status; no policy, no logging

Core Invariants

  1. Resolution failures wake the faulting thread: If a handler encounters an error, the engine wakes the faulting thread rather than leaving it permanently suspended in kernel space.
  2. Exposed Wait Strategies: Blocking (eventfd/poll), BusySpin (low latency, requires dedicated core), and Hybrid (spin then poll).
  3. Atomic WP-Copy Integration: Utilizes UFFDIO_COPY_MODE_WP to atomically map pages write-protected and wake waiters in a single ioctl.
  4. Lockless Latency Histograms: Per-worker log-linear histograms capture detailed percentile tails (p50, p90, p99, p99.9, max) without lock contention.

Building & Testing

Requires Linux 5.7+ (for write-protect mode), CMake 3.16+, and a C++20 compiler. Optional: zlib for DEFLATE compression support.

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build -j
Option Default Effect
UFFD_NATIVE_ARCH OFF -march=native. Off by default for portability.
UFFD_WERROR ON Warnings as errors.
UFFD_BUILD_TESTS ON Build comprehensive test suite.
UFFD_BUILD_BENCHMARKS ON Build all 4 benchmark suites.
UFFD_BUILD_DEMOS ON Build illustrative demos.

Enable unprivileged userfaultfd if running without CAP_SYS_PTRACE:

sudo sysctl -w vm.unprivileged_userfaultfd=1

Running

./build/uffd_tests                              # 48 tests, 1350 assertions
sudo BUILD_DIR=build ./scripts/run_benchmarks.sh # runs all 4 benchmark suites
python3 scripts/generate_benchmark_report.py    # renders benchmark_charts.png
./build/demo_cow_snapshot

Usage Examples

1. Compressed Memory Tier

#include "uffd/codec.hpp"
#include "uffd/engine.hpp"
#include "uffd/handlers/compressed_backing.hpp"
#include "uffd/region.hpp"

// Configure a 1 GB region with a 256 MB resident physical budget
const std::size_t region_bytes = 1ULL << 30;
const std::size_t resident_budget = 256ULL << 20;

uffd::RegionConfig region_cfg;
region_cfg.length = region_bytes;
region_cfg.requested_features = uffd::sys::Features::kPageFaultFlagWp;
region_cfg.register_mode =
    uffd::sys::RegisterMode::kMissing | uffd::sys::RegisterMode::kWriteProtect;

auto region = uffd::Region::create(region_cfg);

uffd::handlers::CompressedBackingConfig backing_cfg;
backing_cfg.resident_budget_bytes = resident_budget;
backing_cfg.track_dirty = true;

auto codec = uffd::make_lzss_codec(); // or uffd::make_zlib_codec(1)
auto handler = std::make_shared<uffd::handlers::CompressedBacking>(codec, backing_cfg);

uffd::EngineConfig engine_cfg;
engine_cfg.worker_count = 2;
engine_cfg.wait_strategy = uffd::WaitStrategy::Hybrid;

auto engine = uffd::Engine::start(std::move(*region), handler, engine_cfg);

// Populate data, then release physical frames to compressed store
// ... write to (*engine)->region().base() ...
handler->compress_and_release_all((*engine)->region().uffd());

// Subsequent accesses are seamlessly decompressed on demand and evicted within budget
const auto stats = handler->stats();
std::cout << "Stored ratio: " << stats.compression_ratio()
          << " Resident pages: " << stats.resident_pages << "\n";

2. Live CoW Snapshotting

auto handler = std::make_shared<uffd::handlers::CowSnapshot>(staging_buffer, length);
// Start engine with MISSING | WRITE_PROTECT ...
engine->region().set_write_protect(true);  // snapshot window opens
// Live writers continue running with minimal microsecond disruption
handler->materialize();                    // complete untouched pages in background

Project Structure

include/uffd/            Public headers (engine, handler, codec, region, mapping, metrics, sys, status)
include/uffd/handlers/   ZeroFill, StagedCopy, CowSnapshot, CompressedBacking
src/                     Engine, mapping, region, sys, codec, handler implementations
src/handlers/            cow_snapshot.cpp, compressed_backing.cpp
tests/                   48 unit and integration tests (test_framework.hpp)
benchmarks/              bench_fault_cost, bench_prewarm_strategies, bench_snapshot_strategies, bench_compressed_memory
benchmarks/support/      TSC calibration, CPU pinning, distribution metrics, CSV/JSON exporters
demos/                   demo_cow_snapshot
docs/AUDIT.md            Retrospective on previous benchmark pitfalls & methodology
scripts/                 Benchmark runners, sweep tools, chart generation

Built With

Share this project:

Updates