Contacted VIMS researcher Ms. Danyang Zhai to share my project and seek expert feedback on its approach to marine debris detection.
Lost fishing gear keeps catching and killing animals for years.
DEPTH reads side-scan sonar, finds lost crab pots (ghost gear) and builds a cleanup plan
DEPTH’s AI agent analyzes hazards, prioritizes inspection routes, and plans recovery actions with human approval.
Analyzes real-GPS sonar data and withdraws unsupported recall guarantees when recordings differ from calibration data.
a person decides the approval tab for the recording.
Connect: plug DEPTH into other tools
aws architecture diagram and how it is designed
Compares DEPTH with manual review to measure the time needed to detect hazards at different recall targets.
Reports test metrics, confidence intervals, and class-level performance, including model weaknesses and unmet targets
DEPTH runs on Graviton-powered AWS infrastructure for robust, efficient, and scalable AI-driven marine debris detection.
DEPTH’s agent validates seabed tracking against depth-sounder data, correcting tracker failures across 280 chunks.
DEPTH integrates with AI agents through MCP, enabling seamless tool connectivity and intelligent, robust marine debris detection workflows.

Inspiration

Every year about 640,000 tonnes of fishing gear is lost at sea. Lost crab pots keep catching animals for years, which is called ghost fishing. In Virginia alone, derelict blue-crab traps killed more than 31,000 crabs, fish, birds and turtles in one six-year study, and removing about 44,000 lost pots returned crab worth around $33.5 million.

The gear sits on the seabed, so it can only be found with side-scan sonar, and today an expert has to review every sonar frame by eye. AI detectors exist, but a cleanup crew can't plan a boat day on a confidence score. I wanted a system that says how much it will find, knows when it can't be trusted, and lets a person decide.

What it does

DEPTH turns a side-scan sonar survey into a cleanup plan that a named person approves.

Finds lost crab pots and wreck debris, in sonar frames or in a raw Humminbird recording with real GPS.
Makes a promise: at least 78% of pots reach a person with one look, and 84% when the agent looks harder, both with 95% confidence: $$\text{pots that reach a person} \ \ge\ 78\% \quad \text{(95\% confidence, fit on held-out recordings)}$$ On unseen recordings it held at 81.1% and 86.0%.
Knows when the promise doesn't apply. On a different sonar it withdrew its promise in 22 of 22 surveys and asked a person for a full review.
Measures the sonar with OpenCV (seabed track, slant-to-ground distance) so map pins land where the gear is, and cross-checks that against the boat's depth sounder.
One review card per object instead of one per box: 3.86 → 2.32 cards per frame.
Plans the cleanup: review queue, routes and re-survey passes. Nothing is dispatched until a named person approves with a PIN, and every decision makes the agent re-plan.
Plugs in: GPX / KML / GeoJSON, an OGC API for QGIS and ArcGIS, and an MCP server so AI agents can use it. Agents can only ask; people decide.

Live demo: https://13-216-140-101.sslip.io · Code: https://github.com/madhesh60/depth

How I built it

OpenCV 5 end to end, CPU only, no PyTorch at runtime.
- Stage 1 measures the geometry with Sobel, blur, medianBlur and remap.
- The YOLO11s detector runs as ONNX through cv2.dnn.
- At start-up DEPTH times both of OpenCV 5's DNN engines and keeps the faster one that gives identical answers. On Graviton4 that cut a frame from 197 to 138.5 ms.
Detector: YOLO11s trained on Kaggle on a cleaned, sonar-only dataset, with whole recordings held out for validation and testing.
Agent: a goal-driven loop: check the promise applies → choose how hard to look → perceive → decide → plan and ask. Every step is logged in plain words.
Statistics: the promises use Clopper–Pearson / Learn-Then-Test bounds. Every study's protocol was committed to git before its unseen data was scored.
AWS: one CloudFormation stack, built for the COOL AMI:
- EC2 Graviton4 with an Elastic IP and HTTPS;
- S3, SSM Parameter Store and Run Command (no SSH);
- CloudWatch, a Route 53 health check, EventBridge;
- SNS approval emails;
- Budgets, Cost Anomaly Detection and a scheduled stop.
App: FastAPI and a zero-build web studio. Over 300 automated tests, lint, type checks and a headless-browser UI test run in CI.
Built solo, with an AI coding assistant (Claude Code) writing much of the code under my direction.

Challenges I ran into

My detector is weaker than published work: F1 0.41 against GhostVision's 0.71–0.73. Six training runs couldn't beat it, because the data is small (about 1,000 unique crab-pot frames). So I changed the goal: don't claim a better detector, make any detector dependable.
Hidden data leaks: the 398 official test images were only 214 unique frames, and there were many rotated copies. I deduplicated everything and held out whole recordings.
Honest statistics: sonar frames from one recording are correlated, which weakened my first 79% promise, so I lowered it to 78%.
Sonar geometry: the image-based seabed tracker failed on some chunks, so the agent now checks it against the depth sounder and re-tracks where they disagree.
Cloud blockers: the COOL Marketplace subscription is blocked by my payment card, and CloudFront is waiting on account verification. The demo runs on Graviton with stock OpenCV 5 for now.
An idea that failed: per-frame "active vision" (the agent choosing its next OpenCV tool) failed its own validation, so it ships switched off.

Accomplishments that I'm proud of

A coverage promise that held on unseen recordings: 81.1% against 78%, and 86.0% against 84%.
An agent that knows when not to trust itself and hands the work to a person.
It runs on a plain CPU: 138.5 ms per frame on Graviton4, and a single core processes sonar about 30× faster than it is recorded.
A live demo on AWS with real human control: PIN-gated approvals, and agents that can only ask.
Every number can be re-run from a script, and the failures are published too.

What I learned

A promise you can check is worth more than a higher score.
Measure before you claim: the faster engine, the correlated frames and the review load all surprised me.
Geometry matters. Without OpenCV's seabed measurement, 151 of 222 pins land outside their own error circle.
Publishing what didn't work saved me from shipping it.

What's next for DEPTH - Debris & Entanglement Prevention and Tracking Hub

Run and publish the COOL on Graviton benchmark once the Marketplace subscription is active.
A timed study with real analysts, to measure the minutes saved, plus a blinded false-alarm audit.
A field trial on a recording with both real pots and real GPS, with a derelict-gear removal programme.
A better detector: it plugs in with one command, and its promise is re-fitted automatically.
On-board use on small boards such as a Raspberry Pi or Jetson.