SiteStamp

AI-powered infrastructure inspection platform for evidence-backed reporting, defect traceability, and historical asset intelligence.

Comment

SiteStamp - Trustworthy AI for Public Infrastructure Inspection

Models used: Gemma 4 31B (gemma-4-31b-it) + Gemini 3.6 Flash (gemini-3.6-flash)

Inspiration

Public infrastructure is only as safe as it is inspected, and inspection capacity, not intent, is the real bottleneck. Bridges, culverts, and roads depend on regular inspection to stay safe, but field engineers are stretched thin. On site, an inspector has to synthesize scattered evidence, photos, handwritten or voice notes, prior knowledge, into one assessment: what's wrong, how severe it is, what should happen next.

In the low-connectivity environments where inspections are common, that synthesis often happens hours or days later, from memory and a blank report template. The result is inconsistent, hard-to-verify documentation, and every hour spent turning scattered evidence into a report is an hour not spent inspecting the next asset. We wanted to build something that closes that gap without cutting the engineer out of the decision.

What it does

SiteStamp turns field visits into evidence-backed engineering reports with traceable defects, severity assessments, repair recommendations, and historical asset context. It's built around an inspection session, not a chat.

An inspector creates a Visit, uploads evidence while walking the site (a photo with optional text or voice notes), then triggers a single Generate Report action. That request sends the complete evidence set to Gemma 4 as one multimodal prompt and returns a structured engineering report: severity assessments, summaries, defect findings, repair recommendations, and supporting evidence for every finding.

The workflow is intentionally bounded. Aside from one optional follow-up if Gemma requests prior inspection history, the model reasons over the entire evidence set in a single pass, the same way an engineer would inspect a site. It can recognize that spalling on multiple piers and a blocked drain point to one systemic issue rather than three unrelated defects, a correlation a per-photo captioning pipeline structurally cannot make, because it never sees more than one image at a time.

That single distinction is the whole project. SiteStamp isn't a vision model bolted onto an upload form. It's a synthesis engine that has to solve three hard problems generative AI is genuinely bad at by default: grounding every claim in verifiable evidence instead of prose, knowing what it doesn't know instead of guessing, and pulling in outside context only when that context is actually load-bearing. Every architectural decision below exists to force those three properties out of a model that isn't naturally reliable at any of them.

Every finding is clickable: click a defect, its supporting photo highlights and scrolls into view. Not "AI said so," verify it yourself. Uncertainty is visible too. When the model can't confirm something, a needsReview banner surfaces it directly instead of burying it in a quiet field.

How we built it

The interesting engineering here isn't the API call to Gemma, it's everything we built to make that call's output trustworthy enough to hand to a maintenance authority. The core design decision was splitting responsibility: deterministic code owns storage, files, and traceability; Gemma owns synthesis and judgment.

Deterministic code always handles storing evidence, visit metadata, and completed reports; resolving Gemma's photo indices back to real records for the UI; validating every model response against a Zod schema before it's persisted; deciding which tool (if any) Gemma is offered, capped at one round trip; and rendering the report and driving the click-to-evidence interaction.

Gemma is responsible for synthesizing multiple photos, notes, and voice transcripts into one coherent assessment; assigning each defect's type, location, severity, description, and which photos support it; flagging its own uncertainty when evidence is insufficient; and deciding whether prior inspection history would improve its assessment.

This split means the UI is never raw model text, and evidence citations are never left to the model's memory of an opaque ID.

Multimodal synthesis. Every evidence photo goes to gemma-4-31b-it as inline image data, alongside the site name, inspector notes, and each photo's note (typed or transcribed). Responses are constrained with the Gemini API's responseSchema and independently re-validated with Zod on the backend, since schema-constrained generation reduces malformed output but doesn't eliminate it.

Evidence-to-defect grounding. This is the citation problem that most "AI report" tools quietly skip. Every defect Gemma emits references the indices of its supporting photos, and the backend resolves those indices to real evidence IDs for click-to-highlight interactions. The report is never a wall of trust-me text; every defect finding can be traced back to the supporting evidence the model was shown.

Agentic history lookup. When an asset code is available, Gemma gets a single tool, get_site_history, and decides for itself whether prior inspections would improve its assessment, rather than history being force-fed into every prompt whether it's relevant or not. The harder part is what the tool refuses to return: results are strictly limited to inspections before the current visit date, at the database query level, not by trusting the model to self-censor. Without that boundary, a later repair could silently leak into an earlier report and corrupt a trend the model reports as fact. When history is used, the report includes a historicalAssessment with trend analysis, a narrative summary, prior inspection count, and a severity-over-time chart.

Self-reported uncertainty. The prompt instructs Gemma to set needsReview: true instead of guessing when evidence is insufficient. During testing, the model correctly flagged that cable saddle condition couldn't be confirmed because no supporting photos existed, rather than inventing a finding.

Two models, two roles. Voice notes are transcribed with gemini-3.6-flash, while Gemma handles the multimodal reasoning. Audio input is currently supported only on Gemma 4's E2B and E4B edge variants, which aren't available through the hosted API we used, so separating transcription from reasoning gave us the required functionality without changing the inspection workflow.

Stack: Node.js/Express/TypeScript backend, PostgreSQL (Neon) with Prisma, React/Vite/TypeScript frontend, local filesystem or Cloudinary for storage (environment-selected), Turborepo + pnpm monorepo, deployed on Render, Vercel, and Neon.

We also deliberately chose the slower model. A side-by-side test of gemma-4-31b-it (55s) against gemma-4-26b-a4b-it (32s) showed the smaller model hallucinating a bridge name it was never given, and dropping an uncertainty flag the larger model raised correctly. Speed lost.

Challenges we ran into

Reliable structured output. An early version had Gemma emit database IDs directly as opaque CUID strings. That significantly reduced JSON reliability, because reproducing arbitrary identifiers inside nested structures turned out to be a surprisingly difficult generation task. Replacing them with small integer photo indices resolved server-side preserved full traceability while improving output reliability.

Audio support wasn't where we expected. A live 400 error revealed that audio support only exists on Gemma's E2B/E4B variants, not the hosted model we were using. We routed voice transcription to Gemini Flash and kept Gemma focused on multimodal reasoning.

We had to choose humility over completeness. Early prompts rewarded Gemma for producing complete reports, which encouraged guesses when evidence was ambiguous. We rewrote the prompt to prefer "I can't confirm this" over a confident wrong answer, which turned needsReview into a reliable engineering signal instead of just a UI flag.

Accomplishments that we're proud of

"Gemma looks at bridge photos" is not what we built. A photo-captioning tool evaluates images independently and has no way to notice that a crack on pier 2 and a crack on pier 4 are the same failure mode. SiteStamp reasons jointly across an entire inspection session, photos, notes, and prior history where available, so it can catch systemic issues a per-image tool is architecturally blind to. That's a synthesis problem, not a captioning problem, and it's the reason nearly every other decision in this project exists. To be clear, SiteStamp assists an engineer by synthesizing the complete evidence set; it does not replace engineering judgment, and the click-to-evidence design exists specifically so a human stays the one who signs off.

We're proud of the trust layer we built around the model rather than just on top of it: structure enforced twice (schema-constrained generation, then independent Zod validation), every finding traceable to a specific photo through an indirection layer we had to design specifically because direct ID generation broke reliability, temporal boundaries enforced at the query layer instead of the prompt, and uncertainty treated as a first-class signal the model is rewarded for using, not an edge case we hope it handles. We're also proud that we chose the slower, more careful model on purpose after measuring the failure modes ourselves, and that we were honest about SiteStamp's own limits, like unauthenticated shareable report links being view-only in this prototype.

Most of all, we're proud that the output isn't a chatbot response. It's a report a maintenance authority could actually act on.

What we learned

We learned that reliability problems in agentic systems often live in unexpected places. It wasn't the reasoning that broke first, it was asking the model to reproduce opaque identifiers inside structured output. Moving that responsibility to deterministic code and giving the model small, easy-to-generate indices instead fixed a whole class of failures.

We also learned that model capability claims need field verification. Assuming audio support followed the model card cost us a live 400 error before we found the real constraint (E2B/E4B only) and adapted.

Most importantly, we learned that for civic infrastructure, trustworthiness has to be demonstrable, not just claimed. That shaped nearly every decision: schema validation on top of schema-constrained generation, an uncertainty flag the model is rewarded for using honestly, and evidence citations a user can click and verify themselves instead of taking on faith.

What's next for SiteStamp

  • Background report generation for evidence-heavy visits. Large visits with dozens of photos can push synthesis time up meaningfully. Moving generation to a background job with progress updates would let inspectors keep capturing evidence on site instead of waiting on a spinner.
  • Streaming model output in the generation view. Rather than a single blocking call, streaming Gemma's synthesis as it's produced, findings and severity assessments appearing incrementally, would make long generations feel responsive and give inspectors earlier visibility into what the model is finding.
  • Flagging and escalating critical findings. Beyond needsReview, a critical severity tier that can be raised either manually by the inspector or as a tool the agent itself calls when it detects an urgent structural risk (like severe spalling on a load-bearing pier), triggering a notification path to a maintenance authority rather than waiting for the full report to be read.
  • Offline-first evidence capture. Since low-connectivity sites are the norm, not the exception, we'd like evidence upload to queue locally and sync when a connection is available, so the field workflow never blocks on network access.
  • Multi-inspector sessions. Larger assets like bridges are sometimes inspected by more than one engineer at once. Merging evidence from multiple inspectors into a single Visit before synthesis would keep the "one coherent assessment" model intact even when the fieldwork isn't solo.
  • Signed, expiring report links. Turning the honest limitation we flagged in this prototype (unauthenticated, view-only sharing) into signed or expiring links, so reports can be shared externally without leaving them permanently open.

Built With

Share this project:

Updates