AfterMath

AfterMath — an agentic post-disaster loss inventory. OpenCV 5 cross-examines your photos; an agent turns the findings into decisions. The photos tell the truth; the agent makes them prove it.

Comment

Inspiration

After a fire or flood, a family faces a strange task: turning a chaotic pile of phone photos into a credible loss inventory for an insurer or relief agency. That pile is hostile evidence — shots are blurry because the lights were out, the same damaged TV gets photographed three times and risks being counted three times, one photo gets submitted under two different claim items, and a claim lists three chairs the photos never fully support. Today a human adjuster untangles this by hand, at the exact moment the family has the least bandwidth. We wanted the computer to do the cross-examination instead.

What it does

AfterMath is an agentic post-disaster loss inventory. A claimant opens a claim and drops in photos. OpenCV 5 analyzes every image, and an agent loop turns those findings into one decision per round:

  • REJECT_PHOTO — blurry or badly exposed evidence, with the reason recorded
  • REQUEST_EVIDENCE — the missing shots, or attribution for untagged photos
  • FLAG_HUMAN_REVIEW — contradictions only a human can reconcile
  • COMPLETE — a reconciled inventory with per-item confidence

The signature moment is the cross-examination: when the same physical view is tagged as two different claim items, ORB keypoints + RANSAC homography prove it's one scene, and the agent escalates to a human gate rather than letting one TV be counted twice. Every round's perception digest and decision is kept as an auditable trace — visual evidence provably changes the next action.

How we built it

Perception (OpenCV 5.0.0 — all substantive analysis):

  • Blur gate: Laplacian variance $\;\sigma^2_L = \mathrm{Var}(\nabla^2 I_\text{gray}) < 50\;$ on a width-normalized decode
  • Exposure: HSV $V$-plane statistics (mean luma + clip fractions at 5/250)
  • Same-view proof: ORB_create (1200 features) → BFMatcher ratio test (0.75) → findHomography RANSAC; a link requires ≥10 inliers and ≥30% inlier ratio — a geometric definition of "same physical view", not a cosmetic one
  • Damage candidates: adaptiveThreshold (Gaussian, 31, C=12) → morphological open/close → findContours; each region scored by local Laplacian-energy deviation, treated as weak evidence only — never a verdict
  • Room grouping: 2D H-S calcHist + compareHist(CORREL) single-link clustering

Decision (agent loop): a validated four-tool action space; the LLM decider (Anthropic/OpenAI/Gemini, swappable) receives the same structured perception digest as the deterministic policy fallback. Invalid or hallucinated tool calls (phantom photo IDs, out-of-range confidence) are rejected against the live session and the policy takes over — recorded in the trace. The demo cannot break because of a provider outage.

Report: per-item confidence $\;c = 0.35 + 0.15\min(v,3) + 0.25\,d\;$ (capped 0.95), where $v$ = distinct views after duplicate collapse and $d$ = damage-candidate score.

AWS (single region ap-southeast-2): API Gateway HTTP API → AWS Lambda (python3.13, arm64/Graviton) where the entire app and every OpenCV 5 operation runs; S3 archives original photo evidence; DynamoDB stores sessions and traces; least-privilege IAM. One-script deploy/teardown, zip-based (aarch64 wheels) — no Docker required.

Challenges we ran into

  • Damage detection honesty. Canny edges miss the soft gradients of real scorch/stain boundaries, and CLAHE equalizes away the very darkness signal we hunt for. The answer was adaptive thresholding on the raw grayscale — but a naive morphological close then welded scattered noise into one full-frame contour. Open→close ordering and an area floor fixed it.
  • "Same view" must be geometric. Ratio-test matches alone aren't proof; RANSAC inlier counts are.
  • Windows PowerShell vs. AWS CLI JSON: inline JSON args get their quotes stripped; every AWS call with structured input now goes through --cli-input-json file://....
  • Anonymous Lambda Function URLs are blocked by this account type's managed policies — solved with an API Gateway HTTP API in front (which is also in the report, honestly documented).
  • Making evaluation reproducible: a procedural scene generator renders each item pixel-identically across views, so duplicate links have ground truth by construction.

Accomplishments we're proud of

  • 15/15 tests against ground truth — including a malicious-decoder test: an LLM that hallucinates photo IDs gets rejected and the policy takes over, traced.
  • 7/7 scripted evaluation sessions complete with all items confirmed (avg 2.4 rounds, ~25 ms of OpenCV per photo) — plus one designed failure case that ends unresolved rather than inventing evidence. The system refuses to fabricate.
  • The entire pipeline runs serverless on Graviton, and cold-starts are handled gracefully.

What we learned

Classical CV is not a legacy option — determinism, explainability, and millisecond latency make it the right tool when a judge (or an adjuster) must audit why a photo was rejected. And an agent is only as trustworthy as its validation layer: the action space, not the model, is the product.

What's next

MCP exposure of the analyzer as a tool for adjuster copilots, batch mode for whole-claim ingest, and a side-by-side human review UI for the flagged contradictions.

Built With

Share this project:

Updates

Submission history