ApexConduit

System Architecture Diagram for "Apex Conduit"

Inspiration: The FP1-to-FP2 Data Prioritization Crisis

A Formula 1 team like MoneyGram Haas F1 operates in a state of constant, high-stakes data analysis. The most crucial period of a race weekend is often the 90-minute window between Free Practice 1 (FP1) and Free Practice 2 (FP2). During this time, a flood of diverse data must be transferred from the congested, high-latency network of a global racetrack (e.g., in Brazil) to the engineering hub in the UK for analysis.

The challenge is not just speed; it's prioritization and predictability.

The data generated is not uniform. It consists of competing streams with vastly different urgencies:

Priority 1 (Critical): A 50MB telemetry snapshot of a critical gearbox sensor anomaly.
Priority 2 (Normal): A 20GB full-session data log for performance modeling.
Priority 3 (Low): A 100GB 4K video file from an onboard camera for archival.

Standard TCP-based tools (FTP, HTTP) are fundamentally incapable of solving this problem.

They fail at prioritization: A large "Low Priority" video file transfer will block the small "Critical" sensor file, a phenomenon known as Head-of-Line blocking.
They fail on instability: On a lossy satellite link, a single dropped packet can stall all transfers, wasting precious minutes.
They fail at prediction: The Strategy Director needs to know, "Will my engineers get the critical file in 2 minutes or 10 minutes?" Standard tools cannot provide an accurate ETA.

This is our specific, mission-critical problem: How do you intelligently transfer multiple, competing files over an unstable link, ensuring that the highest-priority data always gets through first, and provide a reliable, AI-driven prediction of when it will arrive?

This is the problem ApexConduit is built to solve.

The Core Problem: Why TCP Fails

The Transmission Control Protocol (TCP) is the workhorse of the internet, designed for reliability. However, this reliability comes at a cost. TCP guarantees a perfect, in-order stream of data. If a segment #4 is lost in transit, the receiver's operating system will receive segments #5, #6, and #7 but will refuse to deliver them to the application until the missing segment #4 is re-transmitted and received. On a high-latency, unstable link, this creates constant, crippling stalls, destroying throughput.

Our project fundamentally challenges this paradigm for bulk data transfer. We posit that for moving a large file, enforcing strict order at the transport layer is not only unnecessary but detrimental. It is far more efficient to reassemble the file's unordered pieces at the application layer.

Why Not Use an Existing UDP Protocol?

This is a critical question. While protocols like TFTP, UDT, and QUIC exist, they are not optimized for this specific problem statement.

TFTP is too simplistic and lacks modern reliability and performance mechanisms like sliding windows.
QUIC is designed for multiplexing many small web streams and is not tuned for single-stream, maximum-throughput bulk transfer.
UDT is a powerful protocol, but it is a monolithic choice. It has its own prescribed congestion control and does not natively support the application-level Quality of Service (QoS) and Priority Channels that our solution requires. Implementing these features would necessitate deep modification of its core.
Commercial solutions like Aspera FASP prove that a custom UDP-based protocol is the industry standard for high-performance transfers, validating our approach.

By building our own protocol, we gain the granular control necessary to implement our innovative, mission-critical features from the ground up.

Our Key Differentiator: The Synergy of Features

Our innovation is not just in building another UDP protocol. It is in the synergistic combination of three features that no existing open-source tool provides in one package:

Application-Aware QoS: We are not just moving files; we are moving assets with business value. Our protocol understands that a 50MB telemetry file is more important than a 100GB video file and manages bandwidth accordingly.
AI-Driven Predictability: We go beyond a simple (bytes_left / speed) ETA. By using a trained regression model, we provide a reliable prediction that learns from real-world network instability, answering the Strategy Director's actual question.
Deterministic C++ Core: A high-performance engine with no garbage-collection pauses, built for the mission-critical, real-time demands of F1.

Our Solution: A Hybrid P2P Architecture with a Custom Protocol

We are building ApexConduit, a packaged desktop application that uses a sophisticated hybrid architecture to achieve maximum performance. This separation of the Control Plane from the Data Plane is a standard design pattern in high-performance distributed systems, and we apply it here.

The Control Plane (Client-Server): For coordination and session setup, the application uses a lightweight, client-server model. A central Rendezvous Server (or a direct connection) running on Node.js uses WebSockets to handle user discovery, authentication, and transfer permissions. This makes the system easy to use and manage.
The Data Plane (Peer-to-Peer): For the actual data transfer, the system switches to a pure Peer-to-Peer model. Once the connection is approved, the data flows directly between two dedicated C++ engines using our custom protocol. This direct channel offers the lowest latency and highest possible throughput by bypassing any central bottlenecks.

System Architecture: A Deeper Dive

On each user's machine, the application runs as a cohesive suite of three processes, ensuring modularity and performance.

The ApexConduit Engine (C++): The heart of the system. This single, persistent background process is a multi-threaded powerhouse responsible for all data operations. C++ was chosen not just for its raw speed, but for its predictable, deterministic performance. In a system managing precise network timers and buffers, the absence of garbage collection pauses is a critical advantage. This engine handles file fragmentation, packetization, scheduling, reliability, and communication with zero overhead.
The Local Orchestrator (Node.js): The "brain" of the local application. This process serves the UI, manages the application's state, and communicates with the C++ engine via standard I/O (stdin/stdout), sending JSON commands and receiving status updates. It also handles the Control Plane communication with other peers.
The User Interface (React): A modern, responsive web-based UI running in the user's browser. It connects to the local Orchestrator via WebSockets to provide a real-time dashboard of all ongoing transfers, their progress, speed, and predicted ETA.

The Protocol: Apex Transport Protocol (ATP)

At the core of the C++ Engine is our custom application-layer protocol, ATP, built on top of UDP. It implements the features TCP lacks for this specific use case.

Reliability via Out-of-Order Processing: ATP messages are sequenced, but unlike TCP, they are processed as they arrive. This is effective because a file on disk is a random-access medium. If chunk #500 arrives before chunk #499, the receiver's engine can simply seek to the correct offset in the destination file (500 * chunk_size) and write the data. It does not need to wait. This is enabled by an efficient Selective Acknowledgment (SACK) mechanism, which creates a precise map of missing chunks for targeted re-transmission, completely eliminating Head-of-Line blocking.
Session Management & Resumption: A persistent session_id in each message header allows the engine to handle multiple transfers simultaneously. The receiver maintains a bitmap of received chunks on disk, so if a transfer is interrupted, it can be resumed exactly where it left off—a core tenet of resilience.
Verifiable Integrity: We use a two-level integrity check. A fast, non-cryptographic CRC32 checksum is used on a per-message basis to discard corrupted packets in transit. A cryptographically secure SHA-256 hash of the entire file is calculated and exchanged during the initial handshake, guaranteeing that the final reassembled file is bit-for-bit identical to the original.

Innovative Features & How We'll Build Them

Intelligent Prioritization (QoS): The C++ engine maintains multiple priority queues (e.g., Critical, Normal, Low). A weighted scheduling algorithm in the sending thread ensures that high-priority files (like critical telemetry) are always allocated a larger share of the bandwidth than low-priority files (like background logs), regardless of file size.
AI-Powered ETA Prediction: Standard ETA calculations are notoriously inaccurate because they cannot model the complex, non-linear interactions between network conditions. We will build a supervised regression model (a simple neural network) to predict transfer times more accurately.
- Training: We will generate our own training data by running thousands of transfers over a simulated network (tc netem) with varying latency and packet loss. This allows the model to learn from the exact conditions it will face.
- Deployment: The model will be trained in Python with Keras/TensorFlow and exported to the ONNX format. The C++ engine will use the ONNX Runtime library for high-performance, real-time inference, providing users with a far more accurate ETA.

The User Workflow

The Sender and Receiver both have the ApexConduit application running.
The Sender initiates a transfer, selecting a file, a priority, and the Receiver.
The Control Plane handles the permission handshake via the Orchestrators.
Once approved, the C++ Engines are commanded to connect. They establish a direct, P2P UDP data channel.
The transfer proceeds at maximum speed, with real-time status updates pushed to both users' UIs.

Hackathon Scope & MVP

To prove our architecture, our Minimum Viable Product (MVP) for this hackathon will be a functional demonstration of the core protocol, the C++ engine, and the AI-driven prediction.

Our hackathon goal is to build and demonstrate:

The Core C++ Engine: A command-line-only sender and receiver that manages the entire data transfer.
The Core ATP Protocol:
- File Fragmentation: Splitting a single large file into chunks.
- SACK-Based Reliability: Implementing the Selective Acknowledgment (SACK) mechanism. We will prove this works by running a transfer over tc netem with 5% packet loss and showing it completes successfully, while a standard TCP transfer stalls.
Prototype QoS: A proof-of-concept demonstrating that the engine can process a "Critical" priority file before a "Low" priority file.
AI-Powered ETA Integration:
- Training: We will train a regression model on simulated network data (tc netem).
- Inference: The C++ engine will load this pre-trained model (in ONNX format) and use the ONNX Runtime to perform a real-time ETA prediction during a live transfer.

Stretch Goals: The Rendezvous Server and the full React UI are stretch goals. If time permits, we will attempt to build a simple WebSocket bridge from the Node.js Orchestrator to a basic web-based UI to display the transfer status in real-time.

Potential Challenges

NAT Traversal: Real-world peer-to-peer connections are complicated by Network Address Translation (NAT). While our MVP may rely on manual port forwarding, a production-ready system would need to implement STUN/TURN protocols.
Dynamic RTO Calculation: Calculating the optimal Retransmission Timeout (RTO) is a complex art. A static RTO will perform poorly. We will need to implement an algorithm that dynamically adjusts the RTO based on measured Round Trip Time (RTT) and its variance.
Congestion Control: For our MVP, we will focus on maximizing throughput on a dedicated link. A more advanced version would require implementing a custom congestion control algorithm to be a "good citizen" on the wider internet.