Project Story: Places Not Pixels — Accessible Remote Sensing

Team Picasso · OpenCV AI Competition 2026, powered by AWS

Problem statement and intended real-world impact

Satellites photograph every farm, forest and floodplain on Earth every few days, yet answering a simple question — which fields were cropped last year and not this year? which ponds appeared this season? — still takes a bespoke GIS project per question. The bottleneck is the representation: imagery is stored as pixel grids, which cannot be queried, negated or counted.

We turn each satellite pass into a queryable scene: a laminar tree of nested regions (fields, ponds, woods, roads, settlements), each region a token carrying a 64-bit topological address, a class, a confidence and a provenance flag. Tokens are anchored to H3 cells and archived append-only in a bit-column index, so questions compile to processor arithmetic — AND, OR, NOT, population count — and every answer is marked exact or inferred, with vector evidence attached. A scene costs kilobytes instead of megabytes, so the system serves low-bandwidth users: agricultural extension services, water authorities, environmental monitors, insurers.

Built and live: the full pipeline runs client-side in a browser tab at https://huggingface.co/spaces/cp500/team-picasso-opencv-ai-competition-2026 — OpenCV 5 in WebAssembly, a grain U-Net and SAM 2.1 base-plus (int8, 69 MB) under onnxruntime WASM, and a 26M-parameter on-device planner that compiles English questions to exact set algebra. A Landsat scene tokenizes in ~36 s on a laptop with no server round-trip.

Impact target: continuous, auditable change monitoring from open Landsat/Sentinel data at near-zero cost per query, across three use-case families already validated at scale by our prior queryable-EO system (862k cell-period records, three continents):

Grazing and land clearing (rangelands, Landsat 30 m): was this land ever cleared — an exact negative over years, per parcel token.
Smallholder cropping (irrigated delta agriculture, Sentinel-2 10 m): cropped last year and not this year — the class-flip query, at field-token granularity.
Coastal aquaculture (pond mosaics, Sentinel-2 10 m): ponds that exist but hold no licence — new-pond tokens set-differenced against a permit layer. Our synthetic generator already includes a delta-aquaculture archetype, so the pond detector trains on exact ground truth.

OpenCV 5 image analysis (as built)

OpenCV 5 is the symbolic half of a neuro-symbolic tokenizer and does the substantive image analysis at every stage, including inside the browser:

prompt extraction from the U-Net's field-evidence map (threshold → connectedComponentsWithStats), including a second pass that re-prompts areas with moderate evidence left uncovered by accepted masks;
boundary tracing of every SAM mask (findContours, Suzuki–Abe) and Ramer–Douglas–Peucker simplification (approxPolyDP) to produce the vector contours that become tokens;
morphology (morphologyEx) to de-speckle evidence maps;
minAreaRect, geometric moments, min-enclosing-circle roundness, rectangularity and elongation for the exact shape facts;
cross-date change detection: address-aligned token differencing with contour-traced change regions (pure OpenCV, fully auditable);
the COOL benchmark workload: the classical tokenizer and query engine, measured on Graviton vs x86.

Neural proposers supply the regions OpenCV then disposes of: a grain U-Net (8.7 MB ONNX, the segmenteverygrain pattern of Sylvester et al.) proposes one prompt per field; SAM 2.1 hiera-base-plus, dynamically quantized from 277 MB to 69 MB int8, draws one outline per prompt. Both run under onnxruntime WebAssembly in the same tab as OpenCV.js. All proposals are stamped INFERRED; every measurement on a closed boundary is stamped EXACT.

AWS architecture and services (as used)

S3 / open data — Landsat ingest; the Fields-of-the-World training corpus pulled from the AWS open-data mirror (s3://us-west-2.opendata.source.coop/kerner-lab/fields-of-the-world/).
EC2 GPU (g4dn, ap-southeast-2) — training and export box: centroid U-Net training on the FTW corpus, PicoSAM3 fine-tunes (RGB and 8-band multispectral), SAM 2.1 ONNX export and the quantization ladder (fp32/fp16/int8 × tiny/small/base-plus), evaluated head-to-head on two held-out Landsat sites.
Graviton (EC2 c8g) vs x86 (c7i) — the COOL benchmark, run and written up: classical OpenCV parity overall, kmeans 1.8× faster on Arm; cv2.dnn 1.36× slower on Arm because the pip wheel ships without ACL kernels — a reproducible, documented finding, not a hand-wave.
Derived open dataset — the FTW corpus repacked at Landsat 30 m resolution with RGB + 8-band multispectral channels, centroid heatmaps, instance masks and a starter training script, published for other teams: huggingface.co/datasets/cp500/ftw-field-centroids-30m.

High-level architecture (two tiers)

Landsat (open data, USGS)
      │
      ▼
┌─ Tier 1 · server (quality) ──────────────────────────────┐
│ SR ×8 (data-consistent) → grain U-Net prompts →          │
│ SAM 2.1 (full precision) → OpenCV contour tracing →      │
│ laminar tree, 64-bit addresses → calibrated exact facts  │
└──────────────────────────────┬───────────────────────────┘
                               ▼
   Bit-column index:  H3 cell × period × shape address
   (queries = AND/OR/NOT/popcount; exact vs inferred marked)
                               ▲
┌─ Tier 2 · browser (access) ──────────────────────────────┐
│ same pipeline, quantized: grain U-Net 8.7 MB +           │
│ SAM 2.1 base-plus int8 69 MB + OpenCV.js + 26M planner,  │
│ all WebAssembly, ~36 s per scene, zero server calls      │
└──────────────────────────────────────────────────────────┘

Challenges we hit (and what fixed them)

Sim-to-real: the jagged-contour gap. Proposers trained purely on our synthetic world shattered real 30 m Landsat into hundreds of noise-chasing jagged tokens. Sim-trained dense heads did not transfer; SAM-family models plus real labels (FTW) did. We ran a ledger of 31 systematically evaluated tokenizer variants against hand-labelled Landsat ground truth: the score went from 0.58 (baseline) to 0.72 (prompted-SAM redraw) before the grain pipeline made the approach the architecture.
Fact thresholds calibrated on real data, not synthetic. The synthetic world was "too clean": thresholds tuned there misfired on Landsat. Every fact bit (round, rect, ribbon, green, wet) was re-swept against labelled real imagery; perimeter-based roundness (4πA/P²) collapses under boundary jitter, so roundness became area over min-enclosing-circle area, which jitter cannot destroy. The thresholds ship inside the archive so the browser and the Python system read the same numbers.
Laminarity is fragile. Two boundary-sharing regions slipping in as siblings silently zeroed every statistic beneath them. One containment test on interior-pulled points fixed it — and taught us laminarity is the precondition for region statistics being well defined at all.
Quantization vs the browser. onnxruntime-web's WASM backend has no ConvInteger kernel, so default dynamic quantization produces models that load in Python and die in the browser. MatMul-only quantization keeps the size win (ViT weight mass is MatMul) and runs everywhere; the fp16 export path produced type-inconsistent graphs and was dropped. Verified on a ladder: base-plus int8 masks are essentially identical to fp32.
Hypotheses we falsified. A convex-hull rule and a boundary-smoothness gate for round-shape false positives both looked plausible and both failed on rendered probes — the separating feature turned out to be site-specific, i.e. overfitting. We documented the false-positive floor instead of shipping a rule we couldn't defend.
Faint fields. Low-contrast pivots produced no prompt at the standard threshold. Fix: a second prompt pass over areas with moderate U-Net evidence not covered by any accepted mask — recovers them without inflating duplicates (near-duplicate collapse handles the rest).

What we learned

Neural proposes, symbolic disposes. Models propose boundaries and query plans; set algebra computes answers. Because the final answer is CPU arithmetic over bit columns, the system cannot hallucinate a count.
Negation needs a closed boundary. "Green in 2023 AND NOT green in 2024" is native (one AND-NOT and a popcount) once regions have addresses; it is near-impossible over pixel grids or embedding spaces.
Real labels beat more simulation. Fine-tuning on 121k real field instances (FTW) transferred to Kansas/Nebraska Landsat; more synthetic polish did not. Adding an NDVI channel cut false centroid peaks 15→5 and raised same-tile recall 0.40→0.55 — physics channels are cheap wins.
Accessibility is a quantization problem. The distance between "demo on our GPU" and "runs on a teacher's laptop" was: MatMul-only int8, lazy model loading with byte-level progress, and an honest quantization ladder so the quality cost (here: none) is measured, not assumed.

Evaluation method and judge demonstration

Synthetic ground truth: the procedural generator provides exact nested labels; per-level IoU, token-recovery rates and rate–distortion curves of the tokenizer (classical, neural and oracle trees priced by the same DP).
Real-data validation: a self-contained two-site harness (hand-labelled Kansas and Nebraska Landsat, labels and imagery embedded) with a plug-in contract, shared so colleagues can score their own extractors; plus the 31-variant experiment ledger scored on it.
Quantization ladder: tiny/small/base-plus × fp32/int8 evaluated on both sites with mask counts and IoU-vs-reference (collage below).
Judge flow: open the live Space, tokenize a Landsat scene in-tab, click tokens to read their exact situation records, run the bit-column queries (including the class-flip preset), ask questions in English, and watch the refusal behaviour on facts that have no bit column (the licence question). Failure cases documented, including sim-to-real limits.
COOL benchmark: tokenizer latency/throughput/cost on Graviton vs x86, reproducible script included; results in the report.

Featured paths

Both. COOL: the pure-OpenCV tokenizer + query engine on Graviton with a reproducible Arm-vs-x86 benchmark. Agentic Vision: the monitoring agent's perception→decision→action loop, in which OpenCV token output determines the next fetch/zoom/alert action, with full observability at token level.

Figures (upload candidates, paths relative to repo root)

experiments/evolve/renders/quant/quant_ladder_collage.png — the quantization ladder: six SAM 2.1 variants × two Landsat sites, with mask counts and IoU vs reference. Caption: quantization is nearly free; model capacity is not.
experiments/evolve/renders/seg/seg_collage.png — zero-shot grain pipeline across four sites (two US Landsat, two European FTW chips). Caption: the step-change result that set the final architecture.
experiments/evolve/renders/seg/seg_kansasSR_overlay.png — Tier 1: super-resolution ×8 in front of the grain pipeline. Caption: crisp pivots, parcels and the river ribbon at 30 m input.
experiments/evolve/renders/centroid_collage_ndvi.png — the FTW centroid U-Net with the NDVI channel. Caption: physics channels cut false peaks 15→5.
experiments/evolve/renders/unet_picosam_FT_collage.png — U-Net prompts + fine-tuned PicoSAM3 (1.37M params), the earlier browser-tier candidate.
Live Space screenshots: section 2 after tokenizing Kansas Landsat (clean field outlines) and section 4's bit columns — capture fresh from the Space for current numbers.