Road Cleaner

Road Cleaner helps fix the roads by detecting hazards with a four-agent architecture and filing them with your local DOT.

Comment 1

The friction

Everybody drives past the pothole or road debris. It sits there for months, and not because nobody saw it — forty people saw it today. Seeing it was never the problem.

The problem is everything after. Which of sixty-nine agencies owns that specific stretch? Bellevue maintains its own surface streets; WSDOT owns only the state routes running through them, and getting that backwards means a report lands in a mailbox nobody can act from. Which channel does that agency even accept — an Open311 endpoint, a web form, an email? And then the tax on doing any of it: pull over, find the right form, describe where you were, from memory, later.

So nobody does. Google Maps crowdsources sightings and reports them to nobody. The state's own incident feed only contains what the state already knows.

Road Cleaner removes the whole tax. Point a phone at the road and drive. When it finds something it hands you one button, and pressing it produces a located, attributed, correctly-addressed report on the desk of the agency that actually owns that stretch — with a copy in your inbox as the receipt.

What it does

Put the phone in the cradle and press start. From then on nobody talks to it.

It looks every 2.5 seconds — up to six calls to Gemini in flight at once, each stamped with a sequence number so a slow reply cannot draw a box over a road that has already moved on. Every frame is discarded. /api/dashcam/look writes nothing at all: no frame, no detection, no case. Only a find somebody actively presses the button on is kept.

When it finds something, the whole decision arrives at once: the frame the model actually saw, the box drawn on it, the confidence, where you were, and fifteen seconds to decide. Press Report it now and the agent does the part that is actually hard —

  1. Locates it. State by point-in-polygon against shipped boundaries, nearest place from a Census gazetteer in the image. No geocoding API in the loop.
  2. Works out whose road it is. Five deterministic rules first; the model only when all five decline; the case held rather than misfiled if it declines too.
  3. Checks nobody already reported it. Same hazard family, within 500m, in the last 24 hours, across every user.
  4. Composes and sends it on whatever channel that agency actually accepts.

In Bellevue that is a real email to the city's published 24-hour Operations and Maintenance desk. The reporter gets their own copy either way, with the still attached, so there is always a human holding the receipt.

Three refusals, and they are the point

  • No coordinates, no camera. A report without a location cannot be sent to a crew, so the camera will not open until the browser grants location. It is asked for before the camera: requested after, the prompt frequently never appeared on iOS at all, and a whole session would run with every find coming out unreportable.
  • One pothole is one email. Report forty and thirty-nine are saved with a counter and mail nobody. The check reads a projection — hazard type, coordinates, timestamp — never anybody's photograph or address, which is what lets it cross users safely.
  • Whose road it is decides where it goes. A coordinate inside Bellevue reaches the city's own desk; a state route through it reaches WSDOT. Cities declare a service area, and it bounds both the rules and the model's candidate list.

The four agents

The dashcam is one way into a four-agent pipeline. Seven stages:

WATCH → DETECT → CONFIRM → RESOLVE → REPORT → CHECK BACK → PUSH
  • Watcher polls camera sources on tiers — busy corridors every 2 minutes, quiet ones every 10, any camera with an open case every minute — and skips frames identical to the last, capped at ten in a row so a static scene containing a hazard cannot be skipped for ever.
  • Analyst runs vision, then the confidence gate: a 0.55 floor, a second sighting 90s–30min later, a check against the state's own feed within 500m, and a bar that scales with severity (critical 0.60, high 0.70, medium 0.80, low 0.88). A person on an interstate clears a lower bar than debris, because the cost of being wrong is not symmetrical. Pure Python, no model in it.
  • Dispatcher resolves jurisdiction and composes — the same registry and the same words the dashcam path uses. One report writer, not two.
  • Auditor pulls a fresh frame and compares it against the original evidence photo. Cleared closes the case with a before/after pair. Overdue files again one tier up. Overdue twice stops filing and flags a human.

Six of those stages tag a trail entry. WATCH does not, and cannot: polling happens before a case exists, and a trail belongs to a case.

Watch it run: /drill

You cannot wait for a real mattress to fall off a real truck while somebody is judging your project. /drill runs all four agents on demand in about twenty seconds — and it is the honest way to show the architecture, because everything except the location and the imagery is the production code path:

Stage What actually happens
scaffold Gemma 4 turns one sentence into a structured hazard spec
stage Two frames rendered four minutes apart on an invented timeline
detect Gemini 3.7 Flash analyses each frame separately — two real calls
confirm The real confidence gate, same arithmetic as a camera detection
resolve Google ADK works out whose road it is
report The report is composed — and then it stops

The invented camera is given owner_agency_id=None deliberately, so the camera-owner rule cannot shortcut and the ADK path has to actually decide.

It cannot file, and that is the design. A drill case is marked synthetic, excluded from the statistics, invisible to the Auditor, and Dispatcher._file_locked raises rather than returning quietly if anything tries to file one. The drill only ever calls compose(), which the filing channels guarantee is side-effect free. The page shows a Send button that cannot be pressed, and says why.

How Gemini and the ADK are actually wired

Two Google surfaces, both behind ports with a credential-free second implementation, so the whole system runs on a clean clone with no keys.

The dividing line: models judge, code decides

google-adk runs the two things in this system that are genuinely matters of judgement — which agency owns an ambiguous stretch of road, and how a report is worded so a maintenance desk will read it. Everything else — when to poll, whether two frames corroborate, what the SLA is, whether to escalate, whether to file at all — stays in deterministic Python.

That is not squeamishness about models. Control flow that decides whether to contact a government agency should be testable, reproducible, and incapable of being talked into something by a well-phrased frame. The confidence gate is a set of pure functions for exactly that reason, and it is the most heavily tested code in the repo.

ADK, concretely

Two production LlmAgents, both leaves — no tools, no sub-agents:

LlmAgent(
    name="jurisdiction_agent",
    model=self.model,                     # gemini-3.7-flash, a bare model id
    description="Decides which agency owns a stretch of road.",
    instruction=JURISDICTION_INSTRUCTION,
    output_schema=JurisdictionAnswer,     # pydantic, validated on the way back
    output_key="jurisdiction",
)

JurisdictionAnswer carries agency_id, a confidence bounded ge=0, le=1, and a one-sentence rationale. Each call gets its own Runner over its own InMemorySessionService; the final text is scraped off the run_async event stream, keeping only events where event.is_final_response() and the content has parts. Every call is stateless — no conversation history, no cross-case memory. That is deliberate: two questions about the same road should get the same answer.

The instruction is explicit that a hedge beats a confident mistake:

"You are only asked when a rules engine could not decide, so these are the genuinely ambiguous cases. Getting this wrong sends a report to a body that cannot act on it, and the hazard stays on the road while everyone assumes it is handled. […] If none of them is plausibly responsible, pick the closest and set confidence below 0.4 so a human reviews it."

Three independent gates stand between the model naming an agency and that agency being contacted:

  1. json.loads + Pydantic revalidation — an out-of-range confidence is a failure
  2. the id must be in the candidate list that was offered, in the reasoner
  3. the id must resolve in the registry, in registry.resolve()

Any failure at any gate returns None, and None means "the model declined" — which routes to a last-resort rule pass, or holds the case. A missing library, an unset project, quota exhaustion, unparseable JSON and a hallucinated agency id all collapse to the same safe outcome.

The second agent, report_agent, has no output_schema because prose is the product. Its output is discarded and the deterministic draft used instead if it comes back empty, under half the draft's length, or missing the Location: line — on the reasoning that a model which dropped the location has rewritten rather than polished. A failure there costs style and nothing else.

The rules-versus-model boundary

registry.resolve() consults the model at exactly one point, and only after five deterministic rules have all declined:

Rule Matches on
toll-facility road name contains Turnpike / Toll / Expressway Authority
municipal-signal infrastructure damage at an intersection in a listed county
camera-owner the camera states its owning organisation
interstate-mainline road prefix I- / US- / SR- / state routes
city-service-area a bare coordinate inside a city's declared radius

Only then is the model asked. If it declines, a last_resort pass claims state-maintained roads for the state DOT — and a camera on a named road that matched nothing is held, not handed to the state, because a private drive is not the state DOT's problem. The audit trail records which decided: rule_id is either the rule name at confidence 1.0, or reasoner:adk:gemini-3.7-flash at whatever confidence the model claimed.

The agent team, and why its instructions cannot drift

A coordinator with four LlmAgent specialists — watcher, analyst, dispatcher, auditor — mirroring the pipeline stages, exported as root_agent so the whole team can be driven from adk web separately from the loop that calls it.

The detail worth noticing: two of those instructions are f-strings over the same constants the deterministic pipeline enforces. The analyst's instruction interpolates DEDUP_WINDOW_HOURS; the auditor's generates its SLA table by iterating HazardType over DEFAULT_SLA_HOURS. So the agent's description of policy is generated from the policy. It cannot fall out of step with the code the way a hand-written prompt would.

root_agent is built lazily on first attribute access, because adk web imports the module — and so does the test suite, which has no credentials.

Gemini vision, concretely

The client is constructed explicitly for Vertex (genai.Client(vertexai=True, project=…, location=…)), and settings are also exported back into the environment, because ADK builds its own client from the environment when handed a bare model id — without that, ADK silently falls back to the Developer API and bills a different quota.

Three call shapes, all sending raw JPEG bytes inline as a Part:

  • prefilter — a one-word YES/NO screen on a small model, off by default. Fails open: anything that is not "NO" passes the frame through, because discarding a frame is permanent.
  • analyze — one image against a structured prompt, returning hazard type, severity, confidence, description and a box_2d on a 0–1000 grid, converted to fractional coordinates with reversed corners swapped and out-of-frame values clamped. A box the model drew is evidence; there is no invented fallback box.
  • verify_cleared — evidence frame and current frame together, labelled, for the Auditor's before/after.

The error contract is the part that matters: unparseable output raises; "no hazard" returns None. A model outage can never be mistaken for a clear road. Clearance fails the other way on purpose — an unreadable answer means the hazard is assumed still present, because closing a case wrongly stops a real hazard being watched.

Reliability

Both adapters share one retry module: a semaphore outside the sleep, jittered exponential backoff from 1s capped at 16s, and transient detection that checks the exception's type name as well as its text — because ADK wraps quota errors as _ResourceExhaustedError, sometimes with an empty message, which sailed straight through a text-only check as permanent.

Vision gets 4 concurrent slots and 6 attempts; ADK gets 2 slots and 5, because an ADK turn is several model calls behind one await and each slot costs more quota than it looks. A full-speed run before any of this produced 165 consecutive 429s and zero detections — completing successfully and writing nothing, which is worse than crashing.

The live dashcam path opts out of that patience deliberately: asyncio.wait_for bounds it at 20 seconds, because 31 seconds of backoff is right for a scheduler tick and wrong for something a person is watching.

Technologies
Gemini 3.7 Flash (Vertex AI) hazard vision, clearance verification
Google ADK 2.7 LlmAgent + Runner — jurisdiction, report prose, the agent team
Gemma 4 (gemma-4-26b-a4b-it-maas) the drill scaffold: one sentence → a hazard spec
Veo 3.1 (veo-3.1-fast-generate-001) re-stages a confirmed hazard as dashcam footage
Cloud Run the service, scales to zero
Firebase Auth Google sign-in; every incident scoped by verified uid
FastAPI + Jinja2, SQLite, Pillow one deployable artifact, no build step

Every external dependency sits behind a Protocol port with two adapters — one local and credential-free, one Google Cloud — and container.py is the only module that picks. That is why the whole system runs on a clean clone with no API keys, and why going live is an env-var flip rather than a rewrite.

What the live deployment actually runs on, stated precisely because "uses Google Cloud" is doing a lot of work in most submissions:
In use on the deployed service Cloud Run, Firebase Auth, Vertex AI (Gemini + ADK + Gemma + Veo)
Adapters written, not enabled there Firestore, Cloud Storage, Pub/Sub — the deployed revision runs SQLite, local disk and an in-process bus. deploy.sh --with-firestore flips the first two.
Written, not deployed The Watcher and Auditor as scheduled Cloud Run jobs. deploy.sh --with-fleet creates them; the live project was deployed without it. The fleet runs end to end locally via make demo.

Two further models sit in the presentation layer, not the hazard pipeline — separated because they are demo-reel assets rather than product behaviour:
Chirp 3 HD spoken dispatch briefing road-cleaner simulate --narrate, off by default
Lyria (lyria-002) instrumental bed under the demo reel road-cleaner simulate --score, off by default

Lyria writes music and only music — no sirens, no tyre noise. Diegetic sound for a generated clip comes from Veo's own generate_audio. Neither influences a detection, a jurisdiction decision, or a filed report.

Data sources

Public state DOT 511 developer APIs for GA, FL and NC — free, public, and designed for third-party use. Jurisdiction rules covering 69 agencies (62 state DOT, 6 city, 1 toll authority) are hand-built in seeds/agencies.yaml from published district maps and each agency's own intake page. Every contact address was verified against the agency's own site, and an agency that publishes no address is left as a form rather than given a guessed one — three agencies have a real email, and the rest are honest about being forms.

Place names and state boundaries come from the US Census gazetteer, shipped in the image (411 KB) so there is no geocoding API in the loop. No Google Maps content anywhere.

Two imagery sources. Dashcam imagery is a live phone camera, analysed frame by frame by Gemini — that is what the demo video leads with, and what the deployed service serves.

Traffic cameras are a configuration flip. The Vendor511 adapter is parameterised by (base_url, key) and already covers all three launch states, so CAMERA_SOURCE=vendor511 plus a developer key in .env points the Watcher at the live GA / FL / NC feeds. The shipped demo runs on the fixture source instead, and that is a deliberate choice rather than a limitation: it is what lets make demo run the full four-agent pipeline on a clean clone with no credentials of any kind, which is the reproducibility claim this whole submission rests on.

Everything downstream of the frame — the gate, the jurisdiction registry, the report, the refusals — is the same code on both paths.

Findings and learnings

  • Retry policy is not one-size-fits-all. with_retry gives every Vertex call six attempts and up to 31 seconds of backoff — right for a scheduler tick, catastrophic behind a live viewfinder, where one throttled frame froze the page until the phone dropped the connection. The interactive path needed its own, far shorter deadline. Same adapter, same model, different patience budget.
  • Read the logs before tuning. I set that deadline at 9 seconds by feel, then pulled the Cloud Run request logs: successes ran 1.98–8.57s, median 4.7s, and a third of all frames were hitting the cut-off. I had a whole theory about quota throttling; there were no 429s in the logs at all. It was latency with a long tail. Raising the deadline alone made throughput worse — stragglers held their concurrency slots — so the ceiling had to move with it.
  • A browser sends no auth header on an <img> tag. The incidents page fetched its list with a bearer token and then rendered each still as a plain <img src>. Every one came back 401. The tests could not catch it because they all override the auth dependency — a test that stubs the thing that breaks in production cannot see the thing that breaks in production.
  • "Nearest" is not "inside". Routing city streets by nearest place name sent downtown Bellevue to Clyde Hill — a separate city with its own public works department. My own geo module says in its docstring that it means "near X", never "in X". Routing is by distance now, with a conservative radius, because falling through to the state DOT is the safe direction to be wrong in.
  • One real address changes the blast radius of everything upstream. Every city agency was a form pointing at an unreachable endpoint, so a mis-resolution went nowhere and nobody noticed the candidate list was "every agency in the state". The moment Bellevue's genuine maintenance inbox went in, a hazard in Seattle could reach it. The moment one output becomes real, every upstream approximation is promoted to a defect.
  • The gate earns its keep in public. A staged deer produced animal on one frame and pedestrian_on_highway on the other; the gate refused to corroborate them and held the case at watch. Exactly the outcome it exists to produce.
  • A blanket ignore rule shipped a broken image. *.md in .gcloudignore excluded the Gemini vision prompts, which live in agents/prompts/*.md. The image built perfectly and died on startup. There is now a test asserting every runtime-read file survives the ignore rules.
  • The one unprotected model call was the one a person waits on. Every Vertex call in the system goes through the shared retry module — except the drill's opening Gemma call, which had neither retry nor semaphore. A single transient 429 killed the whole drill at stage one, before any of the work worth watching had started. Found by pressing the button.

Built With

Share this project:

Updates