sentinelbrief

This started from a combination that shows up in every transferred segmentation model and almost never in the write-up: precision 0.93, recall 0.38. A model that is right about what it flags and blind to two thirds of what it misses is more dangerous than one that is obviously bad — because it reads as trustworthy. Handing that to a fluent language model and asking "where is it safe?" is the failure I wanted to make structurally impossible rather than merely discouraged.

What it does

SentinelBrief watches a Cloud Storage bucket for new Sentinel-1 radar scenes. When one lands, nobody types anything. A Gemini 3.5 agent on Google's ADK picks the work up by itself and decides how to do it: it inspects the scene's geometry, chooses an analysis window that actually fits inside it, checks whether the model has any measured accuracy for that region, scores its detection against an independent UN (UNOSAT) flood map, and writes the situation brief.

Those are real decisions, not a script. The scene is a fact from the Pub/Sub message; the window, the tool order, and the wording are the agent's. A window chosen for one scene falls off the edge of another — a 10019x4461 scene and a 2048x2048 crop cannot share coordinates — so the agent has to read the geometry before it can choose.

The part that matters is what it will not do. Every number reaching the language model has already passed a deterministic confidence gate, and the gate runs again over what the model writes. It refuses forecasts, water depths, and any statement that an area is safe, because the system provably cannot support them. That is what makes the autonomy above safe to leave unattended: the agent decides what to do, and never decides what is true.

It also decides who to wake up. A finished measurement is not a notification: escalation.py turns the gated assessment into a routing verdict — a response officer for a large detection on a domain the model is measured to be good at, an analyst for anything at low confidence or contradicted by the UN reference, and for a scene where it found nothing at low confidence, nothing at all. That last one is the design decision we would defend hardest. A naive agent sends an all-clear; this one provably cannot, because the model's worst measured recall is 0.38 and silence is therefore not evidence of safety. The refusal is recorded, with that number in it, as the run's own audit trail.

The problem

During a monsoon, somebody repeats the same job every few days: check whether a new radar pass has landed, pull the rasters, clip them, compare against the last pass, and re-type the same report from the resulting numbers. It takes hours, and by the time it is done the information is stale.

The harder problem is what happens when you automate it with an LLM. Our segmentation model, transferred to a flood event it was not trained on, scores precision 0.93 but recall 0.38. That combination is genuinely dangerous: what it flags is almost always real water, so it reads as trustworthy — while it misses roughly two thirds of the flooding. Ask a fluent language model where is safe and it will answer, because answering is what it does.

How it works

One rule drives the architecture: the language model never sees a number that deterministic code has not already ruled on.

• Six tools, all deterministic. The agent chooses among them; it does not compute.
• confidence_gate.py contains no model call of any kind. Same claim plus same tier gives the same verdict forever.
• A claim with no numeric value or no named source is not weak — it is blocked.
• The tier comes from a frozen CSV of measured metrics. A domain the model was never evaluated on returns UNKNOWN, the most restrictive tier. Optimism is never the default.
• The same rule table screens the generated prose as an ADK after_model_callback, substituting [WITHHELD BY CONFIDENCE GATE: <reason>].
• Refusals are never silent. Blocked claims persist to Firestore with their reason and are shown in the console, because a system that hides what it refused cannot be audited.

Autonomy (the Taskmaster part)

The agent makes the decisions, and nobody is in the room. A new COG landing in gs://sentinelbrief-scenes/cog/ fires a bucket notification into the Pub/Sub topic new-scene, which pushes to POST /api/pubsub. That endpoint does not run an analysis. It hands the agent a task — "a scene has landed, analyse it on your own initiative" — and the agent works out the rest:

1. Inspect the geometry. Scenes differ in size, so the window cannot be assumed.
2. Choose a window inside it. This is the decision on show. Nothing in the prompt supplies coordinates.
3. Check whether accuracy is even known for that region. If it is not, the result is unvalidated and the brief must say so.
4. Score against the independent UNOSAT map, if one exists for the event, and report false_negative_km2 — flooding the reference found and this model did not.
5. Write the brief, for an officer who did not request it and does not know the run happened.

The gate is unchanged and still brackets the model on both sides. pipeline.py and confidence_gate.py import nothing from the agent layer, so the autonomy cannot reach the numbers.

It is fail-safe, not fail-closed. If Vertex is unreachable, the run falls back to the deterministic pipeline with a window derived from the scene, and the gated numbers still land. Losing the prose must not lose the measurement. If the agent produces text but never actually analysed anything, that text is rejected rather than published: prose with no measurement behind it is the exact failure this project exists to prevent.

Publishing the feni_2024 scene produced runs 3bca6a43 and adc7cb8f (21.563 km², high confidence, 2026-08-24T21:46) that nobody requested. Nobody clicked a button; the rasters arriving was the entire input.

And here is what the agent decided on its own, verified in production on revision sentinelbrief-00016-9ps (run fe5185ab719a):

• It chose window row 2240, column 8448 on a 10019x4461 scene. Nothing supplied those coordinates. Its stated reason, in its own brief: that is where the UNOSAT reference mapping exists, "enabling pixel-by-pixel accuracy validation." It reasoned from the geometry to a window where its answer could be checked.
• Tool order it picked: inspect_scene, describe_capability_limits, compare_with_reference, analyse_area. It asked what it was not allowed to claim before it measured anything.
• It reported 29.91 km², scored itself against UNOSAT at IoU 0.35, stated that the reference marks 47.3 km² it did not flag, and called its own figure "strictly a lower bound."
• The gate fired on that unattended brief. The agent wrote an operational recommendation containing forecasting and absolute-safety language; it was replaced with [WITHHELD BY CONFIDENCE GATE: forecasting language ('should be'); ... absolute-safety language ('no flooding')]. The honest disclaimer immediately above it — "Do not assume areas without flood flags are safe, dry, or unaffected" — survived untouched.

That last point is the whole design in one artifact: two sentences about safety side by side, the truthful one kept and the unsupportable one refused, on a run nobody requested, with nobody watching. Visible in GET /api/history, which shows the brief and the withheld count for every unrequested run.

One upload is now one run. A scene is four rasters, each landing as its own object, so one upload used to fire four notifications and start four identical runs — which is how the two run ids above came from a single publish. A recency guard on the endpoint suppresses the duplicates and still ACKs them. Honestly stated in the code: that guard is per-instance and reduces duplicates rather than eliminating them. A Firestore lease would be correct across instances, and is deliberately not used, because a dedup miss costs one redundant run whereas a Firestore outage in that path could cost the run entirely.

/api/pubsub always ACKs with 200 and a JSON body, even on a malformed message — an error would make Pub/Sub redeliver a message that can never succeed, an infinite retry loop that burns budget and buries real failures. It must not be a 204: a 204 may not carry a body, uvicorn then raises Response content longer than Content-Length, and Pub/Sub reads that as a 500 and retries forever. It returned 200 to curl and 500 to Pub/Sub, so it looked healthy.

How I built it

Python 3, FastAPI, and Google ADK 2.7.1 on Cloud Run in asia-south1. Gemini 3.5 Flash through Vertex AI in the global location — not us-central1, where that model 404s, which most tutorials hardcode. Keras/TensorFlow for the U-Net, rasterio/GDAL for the imagery, Firestore for state and the audit trail. CI deploys through GitHub Actions using Workload Identity Federation, so there are no service-account key files anywhere.

Challenges

Cloud Run background tasks are CPU-starved by default: 755 s versus 33.7 s for identical work, a 22x difference that looks exactly like slow code. Cloud Run allocates CPU only while a request is in flight, and the analysis deliberately runs in a background task after the response is sent. --no-cpu-throttling fixed it; segmentation alone went from 713 s to 11.4 s.

GDAL cannot authenticate to GCS on Cloud Run without CPL_MACHINE_IS_GCE=YES. It does not consume ADC the way the Python client libraries do, and there is no ADC file on Cloud Run.

/healthz is silently intercepted by Google's front end. The route was registered and visible in /openapi.json, yet returned 404 with no request ever reaching the container. The tell: that 404 arrives without the server: Google Frontend header, whereas a real FastAPI 404 returns {"detail":"Not Found"} with it. The probe lives at /api/health.

Rasterising the UNOSAT reference on demand took 33 minutes per window. Three MultiPolygons carrying 7.9 million vertices; a bounding-box filter did not help, because all three overlap any window. Pre-burning once, offline, into a tiled COG made a windowed read 18.4 ms — about 16,000x faster, identical pixel counts. The 128 MB file made the read look like the bottleneck. It was not. Measure before optimising.

What I learned

315 tests passed while the gate was mangling its own safety caveats in production. The agent is instructed to say "flooding may be missed" — and the gate withheld that exact sentence, because it contains the vocabulary the gate exists to block. Live output read [WITHHELD] [WITHHELD] [WITHHELD] therefore, undetected flooding is likely.... That is worse than no screening: it deletes the most important paragraph in the brief and trains the operator to ignore the marker.

Only running it against the live model found it. Every unit test had used the exact wording of the canned caveats; the real model paraphrases constantly.

The fix taught me more than the bug. After 26 hand-written patterns, each round of probing found more false positives than the last — the signature of fixing symptoms. Two structures explained nine of the last ten: a negated reporting verb ("does not conclude anything about depth") and a safety adjective modifying a material rather than a place ("dry sand", not "dry ground"). Generalising to those took honest sentences wrongly withheld from 23% to none of the 52 sentences I captured from the live model, with no new leaks — and I keep the corpus pinned in the test suite so that number stays checkable rather than remembered.

I also caught myself widening a safety check until it broke: putting ground into the material list let "Dry ground was observed across the district" through. Every "must pass" test in this repo is now paired with a "must still block" guardrail.

Built With

• artifact-registry
• cloud-build
• docker
• fastapi
• firestore
• gdal
• gemini
• github-actions
• google-adk
• google-cloud
• google-cloud-pub-sub
• google-cloud-run
• keras
• numpy
• pydantic
• pytest
• python
• rasterio
• sentinel-1
• shapely
• tensorflow
• u-net
• uvicorn
• vertex-ai