L3S-LEO SciML Engine

• 

  Close-up of the processed Florida Sea Surface Temperature heatmap showing seamless spatial blending.

• 

  Systems engineering layout showcasing the 3-process architecture and independent ZeroMQ data channels.

• 

  Clear documentation and simple terminal commands to compile the native C++ engine and launch the live distributed pipeline demo.

• 

  Monitoring ocean surface temperatures across different global regions by dynamically adjusting spatial coordinates. This is Monterey

• 

  Gulfstream under heavy cloud cover. The C++ engine dynamically handles extreme missing data without dropping frames.

• 

  Raw inputs showing cloud gaps and angle errors (left) processed into a clean, bias-corrected output (right).

• 

  Live dashboard during a 100% persistent storm simulation. The C++ engine handles extreme missing data without stalling or crashing.

• 

  Agulhas Current: Tested at an expanded 1200x800 grid resolution to evaluate performance under heavier spatial matrix loads.

Inspiration

Satellite Sea Surface Temperature (SST) data is fundamentally broken at the point of ingestion. Raw L3S-LEO feeds suffer from asynchronous orbital passes, massive sub-pixel cloud contamination, and slant-path atmospheric distortion. Traditional machine learning and Scientific Machine Learning (SciML) pipelines often fail or learn corrupted physics when forced to ingest this raw noise. I wanted to build production-grade infrastructure that sits upstream of neural networks to reliably clean, weight, and fuse these datasets at high speeds.

What it does

The L3S-LEO SciML Engine is a high-performance, distributed data preparation pipeline. It pulls raw multi-satellite matrices from the NOAA ERDDAP API, securely passes them through a resilient multi-process architecture using ZeroMQ, and executes rigorous continuous weighting algorithms to output a clean, bias-corrected, locked-scale float32 tensor ready for immediate downstream consumption by AI models.

How we built it

• Core Processing Engine: Built natively in C++ to eliminate Python GIL overhead and achieve sub-millisecond latencies. It leverages O(1) Summed Area Tables for rapid window processing and applies continuous exponential weighting (W_i ∝ exp(-S_i / 1.33) * LCR²) to model slant-path geometry (S_i = sec(VZA) - 1).
• Distributed Messaging: Uses ZeroMQ with a blocking PUSH/PULL channel for error-free data-of-record ingestion and a lossy PUB/SUB channel for real-time dashboard telemetry.
• Resilience & Automation: Orchestrated via a robust Bash startup script featuring trap-based cleanup, automatic synthetic data injection during NOAA API timeouts, and a native C++ simulator fallback to guarantee zero pipeline stalls.

Challenges we ran into

Balancing performance with strict memory safety while bridging an asynchronous Python ingestion node with a native C++ processing engine presented complex concurrency hurdles. Managing backpressure across ZeroMQ sockets without causing thread deadlocks or dropping frames during network jitter required precise socket state handling and fault-tolerant fallback logic.

Accomplishments that we're proud of

• Successfully operationalized complex environmental science methodologies into a blazing-fast C++ architecture.
• Engineered a fault-tolerant system that treats network failure as a standard operating condition rather than a fatal crash.
• Delivered a complete, production-shaped data engine complete with a live monitoring dashboard, moving far beyond a typical fragile demo script.

What we learned

Deepened expertise in low-latency systems architecture, inter-process communication patterns, and the critical mathematical preprocessing steps required to feed reliable data into Scientific Machine Learning and Physics-Informed Neural Networks.

Built With

• ai
• bash
• c++
• claude
• cmake
• gemini
• git
• linux
• noaa-erddap-api
• python
• zeromq