Blxrep

• 

  blxrep

Inspiration

Traditional disaster recovery solutions have relied on kernel modules for change block tracking, requiring deep kernel expertise, system reboots for updates, and risking system instability. We envisioned a safer, more developer-friendly approach that could deliver the same sector-level precision without the complexity and risks of kernel module development. eBPF's promise of safe, dynamically loadable programs with comprehensive debugging tools inspired us to reimagine how disk replication could work in modern infrastructure.

What it does

blxrep is a proof-of-concept disk replication tool that leverages eBPF tracepoints to track disk changes at the sector level in real-time. It operates in a dispatcher-agent architecture where:

• Agents run on target servers, using eBPF's block_rq_complete tracepoint to monitor write operations to specific disks without any kernel modifications
• Dispatcher acts as a centralized backup server, collecting both full disk snapshots and incremental sector-level changes over WebSocket connections
• The eBPF program filters write operations to target disks, captures affected sector ranges (start and end), and efficiently communicates changes through ring buffers
• A terminal-based UI (TUI) enables administrators to navigate agents, view backup status, and restore data to any point in time
• Policy-based backup management with configurable intervals, retention periods, and bandwidth limits
• The system performs parallel operations: creating full disk backups while simultaneously tracking live changes, demonstrating the viability of eBPF for disaster recovery scenarios.

How we built it

We built blxrep using:

• eBPF (C): Developed a lightweight tracepoint program attached to block_rq_complete that monitors disk write operations. The program uses eBPF maps to store target disk identifiers (major/minor numbers) and ring buffers to efficiently communicate sector changes to user space
• Go 1.23: Built the entire agent, dispatcher, and TUI application as a single binary for easy deployment
• BoltDB: Embedded database for tracking action states, agent metadata, and dirty sector information
• WebSocket channels: Four dedicated channels (/ws/config, /ws/snapshot, /ws/live, /ws/restore) for real-time communication between agents and dispatcher
• tcell: Terminal UI library for building an intuitive management interface
• The eBPF program performs minimal filtering in kernel space—checking if the block device matches the target and if the operation is a write—then pushes sector ranges to user space where Go handles all business logic, ensuring the eBPF program stays simple, verifiable, and efficient.

Challenges we ran into

1. eBPF Program Complexity vs. Verifier Constraints: Initially, we tried to implement complex filtering logic in the eBPF program, but the verifier rejected our code. We learned to keep the eBPF program minimal and move complex logic to user space.
2. Cloud Platform Block Device Complexity: When testing across different cloud platforms (AWS, GCP, Azure), we discovered that sector-level tracking behaves inconsistently. Sometimes the tracked sectors didn't contain the expected changed data, likely due to virtualization layers performing sector remapping. In other cases, we encountered complex storage stacks where the journey from logical block device to physical disk involved multiple layers of abstraction (LVM, device mapper, virtual disks), making it difficult to track the actual physical location of data. Each cloud provider's storage virtualization implementation added its own challenges, revealing that block-level tracking in virtualized environments is more complex than on bare metal.
3. Handling Network Interruptions: When agents disconnect and reconnect, we initially had no way to track what changed during downtime. Currently, we trigger a full snapshot on reconnection, but we're exploring more efficient incremental approaches.
4. Ring Buffer Sizing: Finding the right balance for the eBPF ring buffer size (currently 1 << 24 entries) to handle high-throughput disks without dropping events required extensive testing under various I/O loads.
5. Sector-Level Granularity: Managing the trade-off between tracking every single sector change versus batching ranges efficiently without missing data required careful design of the event structure and processing pipeline.

Accomplishments that we're proud of

1. Viable eBPF Alternative to Kernel Modules: We successfully demonstrated that eBPF can replace traditional kernel modules for sector-level change tracking, with a verifier-compliant program that runs safely and efficiently.
2. Zero System Disruption: Unlike kernel modules requiring reboots, blxrep dynamically loads and unloads eBPF programs, enabling live deployment and updates on running systems.
3. Complete End-to-End Solution: From eBPF program to distributed architecture to user interface, we built a complete working system that proves the concept works in practice.
4. Real Disk Replication: Successfully replicated actual disk data across networked systems, validating that eBPF-based change tracking captures all write operations reliably.
5. Developer-Friendly Deployment: Created .deb and .rpm packages with systemd integration, demonstrating that eBPF tools can be packaged and distributed as professionally as any traditional software.
6. Intuitive Management Interface: Built a comprehensive TUI that makes interacting with the distributed backup system accessible and visual.

What we learned

1. eBPF Design Philosophy: Keep kernel-space programs minimal and verifiable; push complexity to user space. The eBPF verifier is strict for good reason—safety and correctness come first.
2. Tracepoint Stability: Block layer tracepoints like block_rq_complete provide stable, kernel-version-agnostic interfaces compared to kprobes, making our solution more portable across different Linux distributions.
3. eBPF Maps for Configuration: Using eBPF maps to dynamically configure which disks to monitor (via major/minor numbers) allows runtime flexibility without recompiling or reloading eBPF programs.
4. Ring Buffers for High-Frequency Events: eBPF ring buffers provide excellent performance for capturing high-frequency disk I/O events, with proper sizing being critical to avoid event loss.
5. Community Value: The eBPF community's documentation, tooling (libbpf, CO-RE), and examples accelerated development significantly compared to traditional kernel module development.
6. Virtualization Adds Complexity: Block device tracking in cloud/virtualized environments is significantly more complex than bare metal due to multiple abstraction layers, remapping, and vendor-specific storage implementations. Understanding the entire I/O stack is crucial for reliable sector-level tracking.
7. Proof of Concept Validation: We've proven that eBPF is not just for observability—it's a viable foundation for building critical infrastructure tools like backup and replication systems, though production readiness requires solving platform-specific challenges.

What's next for Blxrep

Immediate Roadmap:

1. Cloud Platform Storage Stack Investigation: Deep dive into how different cloud providers handle block device virtualization to properly map tracked sectors to actual data locations, potentially requiring filesystem-level tracking instead of pure block-level tracking.
2. Intelligent Reconnection Handling: Implement sector-level change tracking during network downtime, allowing agents to send only delta changes instead of full snapshots upon reconnection.
3. Multi-Disk eBPF Optimization: Extend the eBPF program to monitor multiple disks simultaneously using a hash map of target devices, enabling single-agent deployment for servers with multiple volumes. Production-Ready Features (Roadmap):
4. Deduplication: Implement block-level deduplication to eliminate redundant data storage, significantly reducing storage requirements for multiple backups and similar systems.
5. Compression: Add real-time compression for both full snapshots and incremental changes, reducing network bandwidth usage and storage footprint.
6. Encryption: Implement end-to-end encryption for data in transit and at rest to meet security compliance requirements.
7. Advanced Retention Policies: Add tiering strategies (hot/warm/cold storage) and automated lifecycle management for backup data. Future Enhancements:
8. eBPF struct_ops for Block I/O: Collaborate with kernel developers to implement struct_ops support for the block I/O layer, enabling eBPF programs to intercept and modify block device operations directly. This would unlock more sophisticated storage capabilities like inline deduplication, transparent compression, custom I/O scheduling policies, and intelligent caching—all implemented safely in eBPF without kernel modules.
9. CO-RE (Compile Once, Run Everywhere) Support: Migrate to BTF-based CO-RE to eliminate the need for kernel headers on target systems, making deployment even simpler across diverse Linux environments. Performance Metrics via eBPF: Add eBPF-based metrics collection (IOPS, throughput, latency) to provide real-time observability into replication performance.
10. Incremental Forever Backups: Optimize storage by maintaining a single full backup with chains of incremental changes, reducing storage and bandwidth requirements.

Built With

• ebpf
• golang