CivicSense

Inspiration

Every week, city councils across Canada vote on decisions that directly affect residents -- parking fines go up, zoning changes reshape neighbourhoods, budgets get slashed or expanded. But almost nobody reads the meeting minutes. They're buried in obscure government portals, locked inside dense PDFs, and written in impenetrable bureaucratic language. The information asymmetry between local government and the people it serves is staggering.

We asked a simple question: what if an AI agent could read city council minutes for you and tell you what actually matters?

CivicSense was born from the belief that civic engagement shouldn't require a law degree. If we can get an LLM to summarize a research paper, we can get one to tell you that your street's parking fines just doubled.

What it does

CivicSense is an autonomous AI agent that, given any Canadian city name, will:

Search the web for that city's official council minutes repository
Identify the correct government source using Gemini 2 Flash (filtering out news articles, social media, and wrong jurisdictions)
Navigate meeting portals (eSCRIBE, TMMIS, CivicWeb, etc.) to find the latest meeting documents
Download and parse the PDF using IBM's Docling library for layout-aware extraction
Extract motions using Gemini, summarizing each decision into plain language
Display results as a clean card-based interface -- each motion shows its status (Passed/Failed/Deferred), a human-readable summary, category, and impact tags

A user types "Hamilton, Ontario" and gets back cards like:

Parking Fine Increase on King Street -- PASSED Parking fines in the downtown core increased by \$10 effective March 2026. Downtown Residents Parking

How we built it

The system has three layers:

The Agent Pipeline (Python, async):

Scout Agent: Runs 12+ targeted DuckDuckGo queries using a tiered strategy -- first checking vendor portals (eSCRIBE, Legistar, Granicus), then official landing pages, then direct PDF hunting. Gemini 2.5 Flash analyzes the top 15 results with a detailed prompt that enforces location verification and a scoring hierarchy. A heuristic fallback handles API rate limits.
Navigator Agent: Specialized parsers for eSCRIBE (escribemeetings.com) and Toronto's TMMIS system, plus a generic scraper using requests + BeautifulSoup that follows breadcrumb links to find actual documents. Handles FileStream.ashx, GetFile.aspx, and other common government document delivery patterns.
PDF Parser: IBM's Docling converts PDFs to Markdown with layout awareness. Falls back to HTML extraction if the "PDF" is actually a webpage. Validates PDF magic bytes (%PDF) to avoid parsing HTML as PDF.

The Backend (Node.js/Express):

Wraps the Python scraper as a subprocess
Uses @google/generative-ai to extract structured motions from the markdown
Serves the API at localhost:3000

The Frontend (Vanilla HTML/CSS/JS):

Clean card-based UI with search input
Loading state with progress messaging (scraping takes 1-2 minutes)
Motion cards with colour-coded status badges, categories, and keyword tags
Links back to the original source document

The search-to-card pipeline can be expressed roughly as:

[ \text{City Name} \xrightarrow{\text{Scout}} \text{URL} \xrightarrow{\text{Navigator}} \text{PDF} \xrightarrow{\text{Docling}} \text{Markdown} \xrightarrow{\text{Gemini}} \text{Motion Cards} ]

Challenges we faced

The "Wrong Hamilton" Problem: Searching for "Hamilton council minutes" returns results for Hamilton, Ohio; Hamilton, Bermuda; Hamilton County, etc. We had to build strict location verification into the Gemini prompt and the heuristic fallback, explicitly rejecting results from wrong jurisdictions.

Government Sites Fight Back: Many municipal sites use aggressive anti-scraping measures (HTTP 403), broken SSL certificates, or JavaScript-heavy rendering. We implemented browser-like headers, SSL context overrides with retry logic, and specialized parsers for the most common meeting management systems.

The PDF That Isn't a PDF: Some government "download" links return HTML error pages or login screens with a .pdf URL. We added PDF magic byte validation (%PDF header check) and an automatic fallback to HTML content extraction.

Package Dependency Hell on Windows: The docling library pulls in PyTorch, OpenCV, and dozens of other heavy dependencies. Getting clean installs on Windows with Python 3.11 required careful version management, binary wheel preferences, and fallback installation strategies.

The 5-Minute Wall: Some complex portals require multiple hops (search -> portal -> calendar -> meeting page -> document list -> PDF). We wrapped the entire pipeline in asyncio.timeout(300) and ensured every failure path exits gracefully rather than crashing.

What we learned

Municipal data is surprisingly standardized -- a handful of vendors (eSCRIBE, Granicus, CivicWeb) power the majority of Canadian city council portals. Building specialized parsers for these systems unlocked access to dozens of cities at once.
LLMs are exceptional at "needle in a haystack" tasks -- Gemini reliably picks the correct government URL from 100+ search results when given a well-structured prompt with explicit scoring criteria.
Robustness matters more than features -- for a tool that scrapes unpredictable government websites, the difference between "works in a demo" and "works reliably" came down to dozens of try/except blocks, fallback strategies, and input validation.
The last mile is the hardest -- scraping and parsing the data was 30% of the effort; presenting it in a way that's actually useful to a non-technical person was the other 70%.