Inspiration

Municipal decisions are published as long staff reports and council packets. The information residents need is in there (what is changing, who is affected, what it costs, when they can comment), but it is spread over dozens of pages of procedural text. We wanted a resident to be able to upload one of those PDFs and get an answer they can check, not just an answer they have to trust.

What it does

CivicScope takes a municipal PDF and returns:

  • a plain-language summary of what is being decided, by whom and when
  • key amounts and dates
  • what is proposed, who may be affected, potential benefits and trade-offs
  • how and when residents can take part
  • what the document leaves unanswered

Every statement carries a quote and a page number, and links back to that page of the PDF.

How we built it

  • Frontend: React, TypeScript and Vite.
  • Backend: FastAPI. PyMuPDF extracts text page by page so page numbers survive.
  • Model: Claude through the Anthropic API, or a local Ollama model with no API key. Both fill the same Pydantic schema through structured outputs.
  • Verification in code, not in the prompt: after the model answers, the backend checks that each quote really appears on the page it cites. A statement with a quote that cannot be found is removed. A key amount or date must appear character for character inside its quote.
  • Long documents: a PDF that does not fit one request is split by page, each part is analyzed, and a final step selects from the already-verified statements by number, so merging cannot invent new claims.
  • Past consultations: each participation item gets an end date; if that date has passed and the PDF contains it, the item is removed and listed as something to verify.

Challenges we ran into

  • A small local model kept stating a lease from January 2026 to December 2028 as "two years". The quote was correct, so quote checking could not catch it. A stronger model fixed it; the limit of quote checking did not go away.
  • Models wrap quotes in "..." or join two passages into one, which made exact matching reject correct answers. We had to tolerate typography without tolerating changed words or numbers.
  • When splitting long documents, later parts re-analyzed the cover page instead of their own pages until each part was restricted to citing only the pages it was given.
  • A 217-page council packet listed meetings that had already happened as "next steps", because the date was in the model's wording and not in the quote.

What we learned

A matching quote proves the sentence is in the document. It does not prove the interpretation is right. We learned to say that plainly in the interface instead of presenting the output as verified fact, and to test on real documents early: most of the problems above only appeared on real City of Waterloo PDFs, never in synthetic tests.

Results so far

On a 2-page City of Waterloo staff report and the full 217-page council packet it came from, Claude returned 11 and 26 statements with every quote matched to its page, in 14 and 63 seconds. The project has 165 backend tests and browser tests on desktop and mobile widths.

What's next

  • A larger evaluation set across road, zoning and facilities documents, with a human verdict on every statement.
  • Scanned PDFs (OCR).
  • Checking dates and amounts against the municipality's live notices, not only the uploaded file.

Built With

Share this project:

Updates

Submission history