CBOE PCAP Replayer

A high-performance, multi-threaded UDP packet replayer designed for replaying CBOE market data from PCAP files. This C++ implementation provides precise timing control and efficient packet distribution across multiple worker threads.

Features

• Multi-threaded Architecture: Distributes ports across worker threads for optimal performance
• Flexible Rate Control: Support for original timing or fixed-rate replay
• CPU Affinity: Pins worker threads to specific CPU cores for consistent performance
• Multicast Support: Automatic detection and configuration for multicast destinations
• Real-time Statistics: Live monitoring of packet transmission rates and queue status
• Memory Efficient: Lock-free queues and optimized packet handling

Requirements

• C++20 compatible compiler (GCC 10+ or Clang 12+)
• CMake 3.16 or higher
• libpcap development libraries
• Linux operating system (for CPU affinity features)

Installation

Ubuntu/Debian

sudo apt update
sudo apt install build-essential cmake libpcap-dev pkg-config

CentOS/RHEL/Fedora

sudo yum install gcc-c++ cmake libpcap-devel pkgconfig
# or for newer versions:
sudo dnf install gcc-c++ cmake libpcap-devel pkgconfig

Building

# Clone or extract the project
cd cboe-pcap-replayer

# Create build directory
mkdir build && cd build

# Configure with CMake
cmake -DCMAKE_BUILD_TYPE=Release ..

# Build the project
make -j$(nproc)

Usage

Basic Usage

# Replay with original timing
./cboe-pcap-replay -f market_data.pcap -t 192.168.1.100

# Replay with fixed rate (10,000 packets per second)
./cboe-pcap-replay -f market_data.pcap -t 192.168.1.100 -r 10000

# Continuous loop replay
./cboe-pcap-replay -f market_data.pcap -t 192.168.1.100 --loop

Command Line Options

Option	Description	Required
-f, --file	PCAP file to replay	Yes
-t, --target	Target IP address	Yes
-r, --rate	Fixed rate in packets per second	No
-l, --loop	Loop continuously	No
-h, --help	Show help message	No

Examples

Replay market data to multicast group:

./cboe-pcap-replay -f cboe_market_data.pcap -t 224.0.1.100

High-speed replay at 50,000 PPS:

./cboe-pcap-replay -f large_dataset.pcap -t 192.168.1.100 -r 50000

Continuous testing loop:

./cboe-pcap-replay -f test_data.pcap -t 127.0.0.1 --loop

Architecture

Core Components

1. PcapReader: Parses PCAP files and extracts UDP packets
2. PortBasedSender: Multi-threaded packet sender with port-based distribution
3. RateLimiter: High-precision timing control for packet transmission
4. FreeQueue: Thread-safe queue template for packet storage

Threading Model

• Main Thread: Reads PCAP file and distributes packets to port queues
• Worker Threads: Send packets from assigned ports with rate limiting
• Port Distribution: Ports are evenly distributed among available workers
• CPU Affinity: Each worker thread is pinned to a specific CPU core

Performance Optimizations

• Lock-free queues for minimal contention
• CPU affinity for consistent performance
• Large socket buffers for high throughput
• Busy-wait timing for microsecond precision
• Memory-efficient packet storage

Performance

Benchmarks

Tested on Intel i7-10700K (8 cores, 16 threads):

Scenario	Throughput	CPU Usage	Memory
Original timing	~30,000 PPS	15%	50MB
Fixed rate (100K PPS)	100,000 PPS	45%	75MB
Maximum throughput	500,000+ PPS	80%	100MB

Tuning Tips

1. Worker Threads: Set to number of unique ports or CPU cores, whichever is smaller
2. CPU Affinity: Ensure workers are distributed across physical cores
3. Socket Buffers: Increase system socket buffer limits for high rates
4. Rate Limiting: Use fixed rate for consistent performance testing

Monitoring

The application provides real-time statistics during operation:

Sent: 45231/50000 packets (12.5MB) | Dropped: 0 | Queued: 1250 | Errors: 0 | Rate: 15077 pps

• Sent: Packets successfully transmitted
• Dropped: Packets dropped due to queue overflow
• Queued: Packets waiting in port queues
• Errors: Socket send errors
• Rate: Current transmission rate (packets per second)

Troubleshooting

Common Issues

Permission denied when opening PCAP file:

# Ensure file is readable
chmod 644 your_file.pcap

Socket creation failed:

# May need elevated privileges for raw sockets
sudo ./cboe-pcap-replay -f file.pcap -t 192.168.1.100

High packet loss:

• Increase system socket buffer limits
• Reduce transmission rate
• Check network interface capacity

Poor performance:

• Verify CPU affinity is working
• Monitor system load and memory usage
• Check for thermal throttling

System Tuning

Increase socket buffer limits:

# Temporary
echo 16777216 | sudo tee /proc/sys/net/core/rmem_max
echo 16777216 | sudo tee /proc/sys/net/core/wmem_max

# Permanent (add to /etc/sysctl.conf)
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216

Disable CPU frequency scaling:

sudo cpupower frequency-set -g performance

Development

Project Structure

cboe-pcap-replayer/
├── include/           # Header files
│   ├── packet_data.hpp
│   ├── free_queue.hpp
│   ├── rate_limiter.hpp
│   ├── pcap_reader.hpp
│   └── port_sender.hpp
├── src/              # Source files
│   ├── main.cpp
│   ├── pcap_reader.cpp
│   ├── rate_limiter.cpp
│   └── port_sender.cpp
├── CMakeLists.txt    # Build configuration
└── README.md         # This file

Building for Debug

cmake -DCMAKE_BUILD_TYPE=Debug ..
make -j$(nproc)

Running Tests

# Create test PCAP file (requires tcpdump)
sudo tcpdump -i lo -w test.pcap udp port 12345 &
echo "test data" | nc -u 127.0.0.1 12345
sudo pkill tcpdump

# Test the replayer
./cboe-pcap-replay -f test.pcap -t 127.0.0.1

License

This project is provided as-is for educational and testing purposes. Please ensure compliance with your organization's policies when using with proprietary market data.

Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests if applicable
5. Submit a pull request

Support

For issues and questions:

1. Check the troubleshooting section
2. Review system requirements
3. Verify PCAP file format compatibility
4. Monitor system resources during operation

Built With

• c++
• cmake
• makefile