Undercount.

Cities send crews to where people complain. Not to where the problems are.

Comment

Undercount.

Cities send crews to where people complain. Not to where the problems are.

Inspiration

If your streetlight has been out for two years and you don't know you can call 311, or you don't think anyone will come, or you can't do it in English, the city's data says your street is fine.

Not underserved. Fine. There is no problem there at all, as far as the system is concerned.

Cities allocate repair crews based on 311 complaint volume. That rule contains a hidden assumption: that how many people complain tracks how much is actually broken. It doesn't. It tracks who submits. Reporting requires knowing the service exists, believing someone will show up, and being able to navigate a municipal system, usually in English. Those things correlate with income, length of time in the country, and whether you own your home.

So service follows the people who were already best served, and the data makes it look neutral.

This is not a new observation. Kontokosta and Hong documented it in Kansas City in 2021. Minkoff found it in New York in 2016. The finding is solid and it is five years old. What nobody has done is run anything in production that corrects for it.

What it does

Undercount finds duplicate 311 reports so cities count problems instead of complaints.

Eleven people report one pothole in a neighbourhood that knows how to use 311. One person reports one pothole in a neighbourhood that doesn't. The data says eleven problems and one problem. There are two potholes.

Deduplication converts complaint volume into problem count, which is the only fair basis for dispatching a crew. It's also a prerequisite: you can't correct for who's missing until you stop over-counting who's present.

A resident types a complaint in plain language. The system:

  1. Classifies it into the city's real 539-type service taxonomy, with division and section looked up from the city's own mapping rather than generated
  2. Locates it precisely, because the resident supplies an address
  3. Checks it against open cases in a single SQL statement: semantically similar AND within 150 metres AND within 14 days AND same service category
  4. Shows the clerk every score behind the decision, including near-misses labelled with exactly which filter they failed

That last part is the product. A clerk telling a resident "this isn't a duplicate" needs to be able to say why.

Two boundaries, deliberately:

  • Income is never a model input. It's joined once, after every count is final, purely to test whether under-reporting tracks disadvantage. If income were a feature, the system would be allocating on who lives somewhere instead of what's broken there.
  • The output is an inspection flag, not a dispatch. You cannot know from data alone whether a quiet ward has good infrastructure or quiet residents. So the system says "go look here." A human decides.

What we found in Toronto's data

We ran the analysis across all 500,269 Toronto 311 service requests from 2025.

84.32% of published requests have no usable location

Toronto publishes a request's location at one of two precisions: a postal district holding thousands of addresses, or an actual street corner. Never both. 84.32% get the district. 15.68% get the corner.

The city knows exactly where these problems are. They dispatched crews. The precision is stripped before publication because complaint records contain personal information, and every Canadian city does the same thing.

The consequence: you cannot audit a city's service equity from its published data, because the precision required is exactly what gets removed. That is why this has to be software deployed inside the city rather than an outside audit.

We went looking for a demographic effect and didn't find one

We expected higher-income wards to drop in the rankings once duplicates were removed. Only 8 of 25 wards moved, and none by more than 2 places. After excluding data artifacts, only 4 wards move, maximum 1 place.

The reason is arithmetic, not a flaw in the method. Deduplication can only touch the 15.68% of rows with a corner, so it removes 2.68% of the file, and a 2.68% perturbation cannot reorder a list whose adjacent gaps are larger than that.

Income correlation: r = +0.38, p = 0.059, n = 25. The direction matches the literature. At 25 data points it does not clear conventional significance. It is consistent with the research; it is not independent confirmation of it. And duplicate rate does not track income at all, r = -0.10, a flat null. Our original guess was wrong there.

What we found instead was better

In the 15.68% we could examine, one intersection produced 1,948 complaints. Bathurst Street and Canyon Avenue. That is 84.6% of every vehicle noise complaint in Toronto for the entire year.

The timestamps:

17:29:13  17:29:56  17:30:38  17:31:21  17:32:06
17:33:12  17:33:59  17:34:41  17:35:28  17:36:12

43 to 46 seconds apart, for 27 consecutive reports. 30.1% of inter-arrival gaps at that corner fall in the 30 to 60 second band, against 14.7% for every other noise report in the city. That signature appears at exactly two locations in all 500,269 rows.

For comparison, Amplified Sound is the closest analogue and also a noise nuisance. It spreads across 239 intersections with a top-5 share of 10.7%. Bathurst is nine times more concentrated than any comparable category in the city.

These are not 1,948 residents. One source produced them. We can't tell what kind of source from the published data, and neither can the city.

That's the finding. If a single source can produce 85% of an entire complaint category and nobody notices, then complaint volume was never measuring conditions. It was measuring who submits. And that is the number crews get dispatched on.

We also found 1,251 records all geocoded to Metro Hall: 311 feedback about 311 itself, not a condition anywhere. Excluded, with the rationale decided before we looked at the effect.

Both exclusions moved every headline number down, from 26.02% to 22.70%.

Clean stat we'd stand behind anywhere: 34.9% of located pothole reports in Toronto are duplicates. 5,998 requests across 2,798 intersections, top-5 share 2.5%. The opposite of the Bathurst signature, so it's trustworthy.

How we built it

Python and FastAPI, Postgres 17 with pgvector and PostGIS, server-rendered Jinja2 templates. No React, no build step, handwritten CSS. The interface is modelled on the GOV.UK and Ontario design systems because this is municipal operations software, not a landing page.

The dedup core is a single SQL statement with four hard filters and a combined score weighted 0.60 similarity, 0.25 proximity, 0.15 recency. It returns near-misses alongside matches, labelled with which filter they failed, so the decision is auditable.

Complaint text is embedded with text-embedding-3-small. We deliberately do not chunk: 311 complaints average under 200 characters, and chunking would destroy the signal.

Coordinates come from the City of Toronto Centreline Intersection file, with Nominatim as fallback for free-text addresses.

The published dataset contains no complaint text, so demo text is synthetic across ten deliberately distant writing registers, flagged text_is_synthetic in the database and marked in the UI. Service type, location, ward, and timestamp on those records are all real.

Challenges we ran into

Our eval scored perfectly, and that was the problem

Precision 1.000, recall 1.000 on 150 labelled pairs. A perfect score means the test was too easy, so we checked why we passed. Every negative was rejected by a hard filter before the similarity score was ever consulted. The semantic comparison, the part we thought was the centrepiece, was never tested.

So we ablated it. Strip the service-type filter and force the embedding to work alone: best achievable F1 drops to 0.857, and all 25 hard negatives fall inside the cosine distance range of true duplicates. Total overlap. There is no threshold that separates them.

The embedding does not discriminate. The classifier does.

That reframed the whole architecture. The vector index supplies recall, matching "big hole in the road" to Road Pothole / Road Damage across zero shared keywords. The classifier supplies precision, keeping different problems in different types. Neither works alone, and we could only say which was which because we broke our own test.

A geocoder that put an intersection 10.6 kilometres away

Nominatim disagreed with the city centreline by a median of 33.7 metres, but 20 of 100 sampled intersections exceeded 150 metres and the worst missed by 10.6 km. "Coxwell & Danforth" matched a bank branch 1.5 km up the street. A 150-metre filter cannot survive that, and it didn't: recall was sitting at 0.667 purely from geocoding error. The city centreline file is now authoritative for known intersections.

Three more bugs that were silently corrupting numbers

Near-duplicate sets spanning up to 300 metres, violating our own labelling rule. 36 real records seeded twice by independent queries, producing fake duplicates at 0.0 metres and 0.0 days. Degenerate centreline nodes being used as locations.

Confidence that isn't stable

Same input, temperature 0, the classifier's self-reported confidence ranges 0.40 to 0.60 across runs. That number isn't a calibrated probability. It's a token the model generates, and the model has no real introspective access to its own uncertainty. Our confidence floor thresholds a value that moves, which we treat as a stopgap rather than a solution.

Accomplishments we're proud of

We tested it with cases we didn't write. An adversarial set built outside the development loop: 12 hard-negative pairs designed to be similar in wording but genuinely different problems at the same place and time, plus 5 recall pairs in wildly different registers.

Result: 10 of 11 hard negatives held. 5 of 5 recall pairs merged.

The one failure is interesting. Two branch reports, one on the road and one on the sidewalk, both classified Tree Emergency Clean-Up and merged at 0.5879 similarity, higher than every true duplicate in the set. There is no threshold that separates it. The deeper cause: Toronto's taxonomy has no sidewalk-branch category, so the correct answer doesn't exist in the vocabulary. Our system inherits the city's categories, including their gaps.

It knows when to refuse. Below a 0.55 confidence floor the system emits no routing at all: no division, no section, no best-guess field a caller could reach past the floor and use anyway. It returns ranked candidates marked "needs human triage."

Type "my streetlight has been out for two years" and it refuses. Street lighting belongs to Toronto Hydro, which isn't among the 6 of 45 divisions in this dataset. The category genuinely doesn't exist. Rather than confidently misrouting a resident to Parks and Recreation, it says it can't classify this.

24 of 25 abstentions on out-of-scope reports. Zero false abstentions on legitimate ones.

Which also means: the streetlight from our opening line isn't in the published data at all. The invisibility problem exists one layer above the one we're measuring.

Every number got worse when we looked harder, and we published all of them. 26.02% to 22.70% after artifact exclusion. F1 1.000 to 0.857 under ablation. A ward reordering we predicted and then couldn't find. All of it is on the site, next to the exact SQL that produced it.

What we learned

That the most valuable thing you can do with a good result is try to destroy it.

Every finding that survived this project survived an attempt to kill it. The ward thesis didn't survive, and that turned out to be fine, because the reason it failed (84.32% of records have no usable location) is a stronger argument for the product than the thesis would have been. You can't audit a city from outside. That's why the software has to run inside.

Also: a taxonomy is a worldview. Toronto's 311 categories decide what kinds of problems are expressible. There is no sidewalk-branch type and no general streetlight type, so those problems have nowhere to go. Any classifier built on top inherits every one of those blind spots.

What's next for Undercount.

Mississauga first. Their open data stops at daily call-centre volume totals from 2018. They don't publish case-level 311 at all. That gap is the problem, stated plainly.

Then Peel. Brampton and Caledon share regional infrastructure, so it's the same integration twice.

Then the roughly 25 Ontario municipalities over 100,000 residents.

Cities already pay for 311 CRM systems. We attach to an existing budget line rather than creating one. And the deployment model is fixed by the privacy constraint that started all of this: it runs inside their tenancy, the complaint text never leaves the building. The data stays theirs. The software is ours.

Every city added is a population whose unreported problems start getting counted.

Technically: calibrated confidence from logprobs instead of a self-reported number. Retrieval shortlist recall is currently 0.75 to 0.84 and needs work. And validating under-reporting against infrastructure age and repair history, so the inspection flags are grounded in expected condition rather than only in corrected counts.

Try it out

Live: https://undercount.xyz

Code: https://github.com/Greatodu/undercount

Every number on the site is read from evidence.json, produced by running sql/10_evidence.sql against all 500,269 rows, and each result is stored beside the exact statement that produced it.

The intake system needs Postgres with PostGIS and pgvector plus an embedding model, so it runs locally rather than on serverless. make db && make part1 && make serve. The deployed /intake page shows real captured runs of the pipeline rather than a simulation, because a duplicate check without an embedding model and a spatial index is not a duplicate check, and a page pretending otherwise would be the exact failure this project is about.

Built solo in six hours at SaugaHacks 2026 - Ashwin

Built With

Share this project:

Updates

Submission history