Hydra

Inspiration

We come from a sysadmin (Maxime) and a datascience (Cristobal) background, but recently have been working mainly with LLMs. A speech on applied quantitative models by Jack Hidary from SandboxAQ at SCSP's AI+Expo in Washington DC inspired us to diversify since although LLMs perform remarquably well in a very restricted environment, they miss the point altogether on quantitative tasks.

We wanted the challenge of massive datasets, and algotrading felt the sweet spot. As we see ourselves as infrastructure people, we were more interested in working towards a market protection mechanism than a portfolio optimization one, although our project is a base that could be refined to do both.

What it does

We centered our project around the idea of flash crashes, which led us to explore the real time (1s intervals) inter-asset correlations evolution as a predictor. The main challenge is efficient computation of crazy large datasets. In a real trading/market environment, the challenge would be in the number of assets to process in real time. In our project, we focused on vertical complexity with computions running on entire days/weeks of 1s inteval data on 8 Binance pairs simultaneously.

Technicals: The system implements three core computational modules optimized for maximum efficiency on CPU hardware through vectorized operations, parallel processing, and algorithmic complexity management suitable for real-time usage.

Correlation Calculations Correlation matrices utilize rolling window operations with batch processing capabilities. For a given window of T observations across N assets, the algorithm centers data points, computes covariance through optimized matrix multiplication, and normalizes to yield Pearson correlation matrices: C = (1/(T-1)) * (X - μ)^T * (X - μ) ⊘ (σσ^T) All operations leverage Numba vectorization and parallelization. The per-window computational complexity scales as O(TN²), where T << total dataset length, enabling highly efficient streaming computation for real-time applications on multicore CPU architectures.

Spectral Analysis The spectral analysis module tracks the largest eigenvalue of each correlation matrix as the primary indicator of systemic regime shifts and market co-movement patterns. The largest eigenvalue approximation employs the power iteration method with per-iteration cost O(N²) and typical convergence within a fixed number of iterations k=30. Per-matrix spectral complexity scales as O(kN²) with all matrix operations optimized for CPU efficiency. This approach provides robust detection of correlation regime changes that precede flash crash events.

Regime Change Detection Regime changes are computed through weighted fusion of multiple metrics summarizing correlation matrix transitions. The system evaluates normalized Frobenius distance, leading eigenvalue shifts via power iteration, trace changes, and approximate determinant ratios. Batch processing of time-adjacent correlation matrices operates under full parallelization. Overall complexity scales as O(KN²) per window, where K represents batch history length and N the asset count. Statistical normalization through rolling z-scores and persistence filtering requiring multi-period confirmation ensure robust detection without increasing asymptotic computational cost. All computations optimize for CPU performance on Apple Silicon and standard x86 architectures through Numba and NumPy parallelization and vectorization. The system maintains no dependencies on GPU or quantum infrastructure, making it suitable for high-frequency, low-latency financial data streams.

How we built it

We used binance API to get our datasets, then decided we wanted to focus on CPU-optimized calculations, as an incentive for even more efficient computing. (we have in our infrastructure a couple of powerful GPUs, but we wanted the CPU challenge).

It quickly became obvious that given the time we had chosen to allocate, it was either prod-ready real time app with little algorithmic interest, or well thought algorithms lacking prod implementation. We chose algorithms.

Challenges we ran into

We ended up realizing that our project was about the foundation layer of a flash crash predictor - that is, providing the needed real time calculation power - rather than actually building a fined tuned predictor.

With little time and little professional input, fine-tuning parameters was a nightmare. Wish we spent even less time on it, it is not the core of our project. Great lesson for priorization.

Accomplishments that we're proud of

Those calculations run fast. The maths tools we used are advanced, surely their implementation will be useful to someone.

What we learned

At the beginning we explored using Wolfram's new quantum functionalities to speed up calculations. Turns out for this size of datasets, quantum-inspired is no good. Interesting fact.

What's next for Hydra

So much! As previously said, we scaled vertically but not horizontally (only 8 pairs processed). We want to address that by extending to as much assets as we can while keeping computation efficient. Here are our ideas:

Algorithmic Approximation and Dimensionality Reduction: Random projections, sketching techniques, and incremental correlation estimators will estimate large correlation matrices faster while trading minimal accuracy for substantial speed improvements. Low-rank PCA-based approximations and block-diagonalization prove effective when correlations exhibit sectoral or clustered structure, reducing core matrix computational requirements.

Structural Exploitation and Sparsity: When correlation matrices demonstrate sparse characteristics or strong block/hierarchical structure, the system will compute and store only significant entries. This approach scales cost with nonzero or important values rather than naive N² complexity.

Parallel and Distributed Computing: Correlation matrix, eigenvalue, and regime shift computations will distribute across all available CPU cores and, when warranted, across multiple machines. This linearizes scaling for much larger asset universes using standard job queues or distributed computing frameworks.

Windowing and Subsampling: Strided rolling windows computing on every kth window, combined with regime/spectral analysis on adaptively chosen important pairs, enables focus on asset universe regions with maximum change while reducing global computation load.

Incremental and Streaming Updates: The system will update only matrix portions affected by new data rather than recomputing from scratch, leveraging streaming SVD/eigenvalue algorithms to minimize complexity per new sample and maintain continuous processing capabilities.

Advanced Memory Handling: For very large datasets, memory-mapped files and on-disk buffering will avoid RAM overload, with lazy evaluation computing expensive statistics only when required for signal generation.

Built With

• python
• uv