CivicPulse

• 

  CivicPulse

• 

• 

• 

Inspiration

Urban service issues such as water outages, electricity disruptions, mobility problems, and municipal concerns affect everyday life. Yet reports about them can be scattered across data sources and difficult to interpret. We built CivicPulse to turn available public information into clearer, traceable insights that residents and decision-makers can explore.

What CivicPulse does

CivicPulse organizes issue reports by city, issue group, and time. It presents recent counts and Poisson-based estimates, and lets users ask questions through a chatbot. The chatbot uses retrieved project documents and available CivicPulse data to provide grounded answers with source references.

These counts represent classified reports, not independently verified incidents. Forecasts are estimates, and CivicPulse makes that distinction clear.

How we built it

The data pipeline collects information from sources including Free News API, GDELT samples, and Open-Meteo. Raw files and source details are retained in a Bronze layer, then processed through DuckDB into Silver and Gold tables. Gold provides the structured data used for analysis and forecasting.

A Poisson model estimates hourly report counts for supported city and issue combinations. For chatbot questions about report totals or city rankings, CivicPulse queries Gold data directly so the language model does not calculate or invent the results. For guidance questions, a retrieval-augmented generation (RAG) system searches project PDFs, selected official web pages, and CivicPulse documentation. OpenRouter connects the retrieved evidence and data to a language model, which produces a concise answer with references.

Gold counts and forecasts are synced to Supabase. The Streamlit app reads these results and presents the project and chatbot in a user-friendly interface. GitHub Actions can run the scheduled data and sync workflow.

What we learned

We learned that a useful AI answer depends on the quality and coverage of its evidence. Structured data questions are best answered with direct database queries, while document retrieval is better suited to guidance and explanations. We also learned to show the time period, data freshness, and source so users can judge what an answer means.

Challenges

The project’s data can be sparse, inconsistent, or unevenly labeled. This affects city-level comparisons and can produce very small forecast estimates. External sources may also be unavailable, slow, or difficult to fetch reliably. Connecting local data processing, Supabase, scheduled workflows, retrieval, and the deployed app required careful handling of dependencies, permissions, and secrets.

We addressed these challenges by retaining source provenance, distinguishing missing data from zero reports, separating observed counts from model estimates, and instructing the chatbot to say when the available evidence cannot support an answer.

Limitations and next steps

CivicPulse can only describe the sources and locations represented in its data. It cannot confirm that a reported issue occurred, guarantee complete city coverage, or reliably predict every future event. Next steps include improving data coverage and location normalization, evaluating forecast performance as more data becomes available, and continuing to test chatbot accuracy and citations.

Built With

• antigravity
• codex
• copilot
• duckdb
• git-workflows
• github
• numpy
• open-router
• pandas
• pandera
• parquet
• pyarrow
• pytest
• python
• scikit-learn