CivicDiff Packs

Structured civic document analysis powered by Gemini. Turns large document diffs into compact JSON digests with full provenance and validation. Free-tier compatible with demo mode.

Comment

CivicDiff Packs — Full Description

Inspiration

Public documents change constantly — city councils amend ordinances, regulators update compliance rules, and policy shifts happen across jurisdictions. These changes are buried in dense, lengthy documents that stakeholders struggle to track. Missing a critical amendment can mean missed deadlines, compliance failures, or overlooked budget impacts. We built CivicDiff Packs to solve this problem using the power of Gemini AI's structured output and function calling capabilities.

What it does

CivicDiff Packs is a structured civic document analysis platform that turns large document diffs into small, validated, actionable JSON digests — powered by Google Gemini.

The system works through a pack-based architecture where each pack defines:

  • Source document types and safety policies
  • Strict JSON output schemas (validated with Zod)
  • Prompts engineered specifically for Gemini's structured output capabilities
  • Golden outputs for deterministic, reproducible demo mode

The Pipeline

  1. Ingest two versions of a civic document (old + new snapshot, ~30KB total)
  2. Compute a unified diff between them
  3. Analyze the changes using Gemini with structured JSON output and function calling
  4. Validate the output against a strict Zod schema
  5. Self-check the result with a second Gemini call for correctness, safety, and schema compliance

The output is a compact digest (~2KB) containing:

  • Executive summary of all changes
  • Structured change entries with impact levels (high/medium/low)
  • Deadlines extracted from the documents
  • Action checklist for affected parties
  • Risk flags for items requiring attention
  • Full provenance linking every finding back to exact source text

Included Packs

  1. City Council Minutes (EN) — Tracks municipal ordinance changes, zoning amendments, budget approvals, and council votes
  2. Regulation Update (ES→EN) — Analyzes Spanish regulatory documents and produces English digests with original terminology preserved

Try Your Own Data

Judges and users can download sample packs, modify them with their own documents, and re-upload to analyze. The Pack Manager validates structure before upload and the pipeline seamlessly processes user-provided content.

Gemini API Logger

A built-in toggleable logging system captures every Gemini API call in real-time:

  • Raw request/response payloads with full transparency
  • Token estimates (input + output) for cost tracking
  • Timing and latency metrics for performance monitoring
  • Error traces with retry information for debugging
  • Export to JSON for offline analysis and compliance

This provides full visibility into what Gemini is actually producing — critical for trust, debugging, and cost management.

Visual Analytics Dashboard

Interactive charts powered by Recharts show real-time insights:

  • Impact Distribution — Donut chart of high/medium/low changes across all analyses
  • Action Priorities — Bar chart of P0/P1/P2 items for immediate attention
  • Performance KPIs — Schema pass rate, response time, token usage, system uptime
  • Budget Tracking — Live API call counting and hourly limits

Pack Manager & Extensibility

Upload/Download System for complete pack customization:

  • Download sample packs as JSON bundles for offline study
  • Upload custom packs with your own documents and prompts
  • Automatic validation of pack structure (7 required files)
  • Dynamic pack discovery — uploaded packs appear automatically in the registry
  • Perfect for organizations wanting to analyze their own document types

Advanced Error Handling & Resilience

  • Automatic Retry with exponential backoff (2s→4s→8s) for 503/UNAVAILABLE errors
  • Evidence Truncation prevents schema validation failures when Gemini returns >5 evidence items
  • Rate Limiting protects API keys with configurable per-IP limits (5/min live, 20/min demo, 50/hour total)
  • Schema Validation with Zod ensures all outputs conform to strict JSON schemas
  • Session-based Reports for sharing analysis results with team members

Gemini API Features Used

  • Structured Output: Every Gemini call uses responseMimeType: "application/json" with responseSchema to guarantee valid-by-construction JSON output conforming to our Zod schemas
  • Function Calling: 4 tool definitions (compute_diff, validate_candidate_json, extract_provenance, persist_report) in an agentic pipeline loop
  • Self-Check Validation: A second Gemini call validates the primary output for JSON validity, schema compliance, word limits, and safety
  • Large Context Processing: ~30KB of document input condensed into ~2KB of structured, validated output
  • Model Selection: Users can choose between free-tier models (Gemini 2.5 Flash Lite, Flash) and paid models (Pro) directly from the UI

Two Modes

  • Demo Mode — No API key required. Uses pre-computed golden outputs for instant, deterministic results. Perfect for evaluation and testing.
  • Live Mode — Real Gemini API calls with structured output, self-check validation, token estimates, and hourly budget tracking. Auto-retry with exponential backoff for transient errors.

Tech Stack

  • Framework: Next.js 16 (App Router, TypeScript) with Turbopack for fast builds
  • Styling: Tailwind CSS + shadcn/ui components for modern, accessible UI
  • AI: Google Gemini API (@google/genai) with structured output and function calling
  • Validation: Zod with strict schemas for runtime type safety
  • Testing: Vitest with 31 passing tests covering core functionality
  • Deployment: Vercel / Docker with environment-based configuration
  • Package Management: pnpm for efficient dependency management

Production Ready Features

Security & Reliability

  • Rate Limiting: Configurable per-IP limits via environment variables
  • Budget Protection: Hourly API call caps prevent runaway spending
  • Content-Type Validation: API routes reject malformed requests
  • Security Headers: X-Content-Type-Options, X-Frame-Options, Referrer-Policy
  • CI/CD: GitHub Actions with dependency audit and secret scanning

Performance & Monitoring

  • Real-time Logging: Toggleable Gemini API call logger with export capabilities
  • Health Endpoints: /api/health returns system status, pack count, and available features
  • Error Tracking: Comprehensive error logging with retry information
  • Token Estimation: Input/output token counting for cost management

User Experience

  • Responsive Design: Mobile-first UI with dark/light theme support
  • Progressive Enhancement: Demo mode works without API key, live mode when available
  • PDF Export: Formatted report downloads via print-to-PDF
  • Session Management: Report sharing through session-based URLs
  • Accessibility: Full keyboard navigation and screen reader support

What Makes This Different

Unlike generic document analysis tools, CivicDiff Packs is purpose-built for civic documents with:

  1. Domain-Specific Prompts engineered for municipal and regulatory language
  2. Bilingual Processing preserving original terminology (ES→EN)
  3. Provenance Tracking linking every finding to exact source text
  4. Impact Classification (high/medium/low) for prioritized action
  5. Deadline Extraction with automatic compliance reminders
  6. Risk Flagging for items requiring immediate attention
  7. Structured Output guaranteed valid JSON for downstream processing

The combination of Gemini's structured output, function calling, and self-check validation creates a reliable, production-ready system for turning document chaos into actionable intelligence.

How We Built It

Architecture

CivicDiff Packs is a Next.js 15 fullstack application using the App Router. The architecture separates concerns into three layers:

  • src/lib/server/ — Backend logic: Gemini API integration, pipeline orchestration, rate limiting, diff computation
  • src/lib/client/ — Frontend utilities, mock data, UI helpers
  • src/lib/shared/ — Zod schemas shared between server and client for validation

The Pipeline

We built a multi-step analysis pipeline that processes document changes:

  1. Pack Loading — Each pack is a self-contained directory (packs/{id}/) containing prompts, fixtures, schemas, safety policies, and golden outputs
  2. Diff Engine — Uses the diff library to compute unified diffs between document snapshots
  3. Gemini Structured Output — The core analysis call uses responseMimeType: "application/json" with a responseSchema that maps to our Zod digest schema. This guarantees valid-by-construction JSON output
  4. Function Calling (Agentic) — An alternative pipeline mode uses 4 tool definitions in a multi-turn agentic loop: compute_diff, validate_candidate_json, extract_provenance, persist_report
  5. Self-Check — A second Gemini call validates the primary output for JSON validity, schema compliance, word limits, and safety policy adherence

Gemini Integration

We use the @google/genai SDK with:

  • Structured Output schemas defined using @google/genai's Type enum, mapping directly to our Zod schemas
  • System instructions loaded from per-pack markdown files
  • Temperature control (0.2 for analysis, 0.1 for self-check) for consistent outputs
  • Thinking level (medium/high) as a user-configurable parameter
  • Automatic retry with exponential backoff (2s→4s→8s) for 503/UNAVAILABLE errors

Frontend

  • shadcn/ui components for a polished, accessible UI
  • Real-time pipeline visualization with step-by-step progress, logs, and timing
  • Model selector letting users choose between free-tier and paid Gemini models
  • Rate limiting UI showing hourly budget remaining
  • Report viewer with digest sections, raw JSON, evidence chips, and export options

Safety & Quality

  • Zod validation on every output — malformed results are rejected
  • Per-pack safety policies enforced via system prompts
  • Rate limiting (5 calls/min, 50/hour) to protect free API keys
  • Security middleware with CSP headers and Content-Type validation
  • CI pipeline with dependency audit and secret scanning # What We Learned

1. Gemini Structured Output Is Powerful but Requires Careful Schema Design

Using responseMimeType: "application/json" with responseSchema eliminates the need to parse free-form text, but the schema must be carefully designed:

  • Too many constraints confuse the model; too few produce inconsistent output
  • Word limits and array length constraints work best when reinforced in both the schema description AND the system prompt
  • The Type enum from @google/genai maps cleanly to JSON Schema but requires explicit property definitions

2. Function Calling Enables True Agentic Workflows

Gemini's function calling goes beyond simple tool use. With 4 tool definitions in a loop, we built a genuine agentic pipeline where the model decides which tools to call and in what order. Key learnings:

  • Always set iteration limits to prevent infinite loops
  • The model generally follows the tool sequence described in the prompt, but needs guardrails
  • Function responses should be concise — the model processes them as context for the next step

3. Self-Check Is a Game Changer for Trust

Adding a second Gemini call to validate the primary output dramatically improves reliability:

  • It catches edge cases like exceeding word limits or missing required fields
  • The self-check prompt should be specific: check JSON validity, schema compliance, word limits, and safety
  • Low temperature (0.1) for self-check produces more consistent validation results

4. Free-Tier Requires Defensive Engineering

Building for Gemini's free tier taught us about resilience:

  • Models can be overloaded (503) at any time — automatic retry with backoff is essential
  • Rate limits vary by model and time of day
  • Offering model selection in the UI gives users agency when one model is saturated
  • Budget tracking builds trust by showing users exactly how many calls they have left

5. Demo Mode Is Essential for Evaluation

Having a fully functional demo mode that works without an API key is crucial:

  • Judges and evaluators can test immediately without setup
  • Golden outputs serve as regression tests — if the schema changes, golden outputs must still pass
  • Demo mode also serves as documentation of expected output quality

6. Pack Architecture Scales Naturally

The pack-based design (each pack = directory with prompts, fixtures, schemas) turned out to be extremely practical:

  • Adding a new document type requires zero code changes
  • Per-pack safety policies prevent cross-contamination between document types
  • Golden outputs per pack enable targeted testing # What's Next for CivicDiff Packs

Near-Term Roadmap

More Packs

  • Federal Register (US) — Track changes to federal regulations and proposed rules
  • EU Directive Updates — Monitor European regulatory changes with multilingual support
  • State Legislature Bills — Track bill amendments across US state legislatures
  • Corporate Policy Updates — Extend beyond civic documents to internal compliance

Real-Time Monitoring

  • Webhook Integration — Trigger analysis automatically when document sources update
  • Scheduled Scans — Periodic polling of document repositories for new versions
  • Email/Slack Notifications — Alert stakeholders when high-impact changes are detected

Enhanced Analysis

  • Multi-Document Comparison — Compare more than two document versions simultaneously
  • Trend Analysis — Track how specific regulations evolve over time across multiple updates
  • Cross-Pack Correlation — Identify when changes in one jurisdiction affect another

Long-Term Vision

Community Pack Registry

  • Public Pack Marketplace — Community-contributed packs for different document types and jurisdictions
  • Pack Templates — Starter templates to make creating new packs even easier
  • Version Control — Git-based pack versioning with automatic golden output regeneration

Advanced Gemini Integration

  • Grounding with Google Search — Enrich analysis with real-time context from web sources
  • Multimodal Analysis — Process scanned PDFs and images of documents using Gemini's vision capabilities
  • Streaming Output — Stream analysis results in real-time as they're generated

Enterprise Features

  • Team Workspaces — Shared pack registries and report history across organizations
  • Audit Trail — Complete history of all analyses with version tracking
  • Custom Schemas — Let organizations define their own output schemas for specific compliance needs
  • On-Premise Deployment — Docker-based deployment for organizations with data sovereignty requirements

Built With

Share this project:

Updates