SeaPredictor

Inspiration

We were inspired by how the model unifies traditionally separate tasks like perception, georeferencing, and planning into a single coherent cycle where observations influence state, which in turn influences action.

What it does

SeaPredictor is an end-to-end system that detects marine debris and forecasts its trajectory

For detection we used a ResNet-18 CNN, trained on the MARIDA (Marine Debris Archive) dataset, scans Sentinel-2 satellite tiles (11 spectral bands) to identify 15 classes of objects, including marine debris, sargassum, foam, and ships.

Then for forecasting we used OpenDrift (a Lagrangian particle simulator) and NOAA OSCAR ocean current data, the system propagates detections forward in time. It simulates how debris will drift over a 1–30 day horizon.

Lastly it visualizes the forecasting and prediction using CesiumJS 3D globe web app allows users to pick historical satellite scenes, view detected "hotspots," and run on-demand Monte Carlo simulations to see predicted particle clouds and drift paths.

How we built it

We adapted a ResNet-18 architecture to handle 11-band satellite imagery by tiling and rescaling pretrained ImageNet weights. We implemented per-class threshold tuning to maximize the Macro-F1 score (reaching 0.85 for Marine Debris).

We integrated the OpenDrift library, forcing it with daily surface current vectors from NOAA’s OSCAR dataset. We built a custom data pipeline (oscar_concat.py) to handle coordinate transformations and temporal alignment.

Challenges we ran into

Class Imbalance: Marine debris accounts for only ~0.004% of pixels in the training data. We solved this using BCEWithLogitsLoss with automated per-class weighting and custom thresholding.

Temporal Drift: Our original plan to use an LSTM to fuse current data into the detection stage actually regressed our accuracy. We had to make the difficult engineering call to pivot to a "CNN-only" detection model and keep the physics engine as a separate stage.

UI/UX: We had a lot of issues tying the backend and frontend together.

Accomplishments that we're proud of

Getting the models to predict where the trash will be, and actively rendering the outputs from the models into an interface users can see.

What we learned

Translating academic concepts, specifically the Spatio-Temporal Road Image Dataset and World Model for Autonomy, and applying it to a whole different project.

Citations

Carrión, H., Bai, Y., Hernández Castro, V. A., Panaganti, K., Zenith, A., Trang, M., Zhang, T., Perona, P., Malik, J., & Tera AI. (2025). TARDIS STRIDE: Spatio-Temporal Road Image Dataset and World Model for Autonomy. arXiv preprint arXiv:2506.11302v3.

What's next for SeaPredictor

Built With

• cesiumjs
• next.js
• numpy
• opendrift
• pandas
• python
• pytorch
• scikit-learn