Inspiration
Mississauga is one of the most linguistically diverse cities in Canada. In the 2021 Census, 24,305 residents (3.4%) could not hold a conversation in English or French, and tens of thousands more speak Urdu, Punjabi, Tagalog, Tamil, Arabic or Mandarin most often at home. City services are increasingly answered by voice AI. When a resident speaks to a 311 line, their words pass through speech-to-text, then machine translation, then a staff member. Every step can change the message, and the usual way to judge these systems is whether the English "reads well." But a request can read perfectly and still lose the one thing staff act on: the address, the date, the number, or the word "not." Nobody measures where that happens.
Our question: when a resident speaks to a voice AI in their home language, which critical facts survive to the English staff read, which step loses them, and does telling the system the language up front help?
What it does
Lost in the Line tests the full voice line against a built-in answer key: the same 40 sentences read aloud by real speakers in 14 languages including English (560 recordings). For every clip we know exactly what the speaker meant, so we can measure, fact by fact, what survives.
It is a 4-screen prototype a judge(ML pipeline/agent) can use in 60 seconds:
- Play the Line - read a real sentence in its language (or play a verified recording). A trace shows the detected vs expected language, what the ear heard, the English staff receive beside the human reference, and each critical fact as survived, changed or missing. A Benchmark Receipt shows that language's precomputed evidence.
- False-Comfort Matrix - the headline. Every clip placed in a 2×2: did the AI meaning check say "fine," and did the critical facts actually survive? The dangerous cell is "looks fine, a fact was lost."
- Power-Up Decision Lab - switch on "ask the language first" and see the measured, paired change; plus "would a confirm-back have caught this?"
- Mississauga - neighbourhoods ranked by benchmark-weighted language exposure (a screening index, not a forecast), each with how much of its population our tested languages cover.
Every screen has a Methods drawer with sample sizes, intervals, model and prompt versions, and limits. Every number on screen traces to a results file, and every results file to code.
What we found
- The ear can write a caller down as the wrong language, invisibly. Scribe labelled 7 of 40 Urdu recordings as Hindi (in Devanagari script) and wrote 22 of 40 Cantonese recordings in Mainland Simplified instead of Hong Kong Traditional script. English: 0 of 40. When the language and script were right, character error rate was 0.026; when wrong, 0.543. The English translation often still reads fine, so a human reading it would never notice.
- A request can pass the meaning check and still lose a critical fact. 20 of 322 fact-bearing recordings (6.2%, 95% CI 2.8%-9.6%) were graded "same or minor" by the AI meaning check yet dropped a number, date, name, or the word "not." In 17 of the 20 the loss was at translation; in 3, at the ear.
- The fix depends on the language. Supplying the language up front took Urdu's correct-label rate from 82.5% to 100% and its error rate from 0.215 to 0.036, but Cantonese's correct-script rate only from 40% to 52.5%, and critical-fact survival did not measurably change.
- A confirm-back step would catch most silent errors. For 24 of the 28 recordings with a wrong fact, reading that fact back to the caller would have surfaced it.
- Our own AI judge is too lenient to trust alone. In a hand-check of 20 translations, the judge agreed with the human reviewer on 12 of 20, and every disagreement was the judge calling a changed message "fine" - which is exactly why we check facts by rule instead.
How we built it
1. The ear. Every clip ran through ElevenLabs Scribe v2 with automatic language detection, which is what a real line faces. We score character error rate (chosen because Urdu, Punjabi and Tamil have inconsistent word boundaries and Chinese has none), and track language and script detection separately. Every response is cached for reproducibility.
2. Translate twice. Each clip is translated to English with Google Gemini (gemini-3.8-flash, temperature 0) twice: once from the clean transcript, and once from what Scribe heard. Comparing the two lets us label each fact as survived, lost at the ear, or lost at the translator — without pretending the split is a clean subtraction.
3. Critical facts, by rule, not AI. From the English answer key we extract numbers (including "lakh"/"crore"), dates, negation, and names, with deterministic rules, and check whether each still appears in the translation. We deliberately do not use AI to check whether AI lost a fact.
4. Three meaning signals. An AI judge (Gemini, fixed rubric), embedding similarity, and the rule-based fact check. The "false comfort" metric fires when the first two say "fine" but the facts say otherwise.
5. Check the checker. A reviewer hand-labelled 20 random pairs (the judge agreed on 12 of 20, always erring lenient). Every per-language number carries a paired cluster bootstrap 95% interval (resampled by sentence, 2,000 reps).
6. From clips to neighbourhoods. For neighbourhood $n$, with $P_{n,\ell}$ residents speaking tested language $\ell$ at home, measured critical-fact loss rate $R_\ell$, and language-base population $P_n$:
$$E_n = \frac{\sum_{\ell} P_{n,\ell}\, R_\ell}{P_n}$$
We report coverage (the share of residents our 13 tested languages reach) and call it a screening index, not a forecast.
7. The fix, measured. We re-ran the three worst languages with the language supplied to Scribe, paired per clip, and modelled a confirm-back that reads critical facts back to the caller.
8. Context. We cleaned Mississauga's 311 call-centre data (two file formats, typo dates, holidays stored in a numeric column, 44 blank rows): about 1,006 calls per business day at ~3.4 minutes each.
We did not train a model. We evaluated production AI against a 14-language answer key, and checked our own evaluator.
Challenges we ran into
- Separating the ear from the translator. One translation hides which step failed; translating twice isolates them.
- Fair comparison across scripts. Word error rate punishes Tamil and Urdu and does not apply to Chinese, so we used character error rate as the common measure, and treated "right language, wrong script" as its own error (which is how we found the Cantonese problem).
- Trusting our own judge. Gemini translates and judges, so we added embeddings, rule-based fact checks, and a human spot-check.
- Free-tier limits. Gemini's free tier allows about 20 requests a day, so we batched up to 200 translations per request with a recursive split-on-failure, bringing a full run under the cap. One model was closed to new accounts mid-event, so we moved everything to one model and re-ran so no result mixes models.
- Honest fact rules. Our first rules counted correct paraphrases as losses ("not uncommon" read as "common," "20 lakh" vs "two million"). We fixed each, unit-tested it, and every fix removed a false alarm rather than adding one.
Accomplishments that we're proud of
- An evaluation that tells the City which step to fix, per language, not just that something is wrong.
- A metric - "false comfort" - for the failures a human reader would never catch.
- Checking our own evaluator three ways, including a human review that it failed in the lenient direction.
- A fix we measured, and a reproducible, cache-first pipeline that re-runs offline at zero cost.
What we learned
- "Accuracy" hides what matters: a translation can sound right and still drop the one number that sends a truck to the wrong street, or the "not" that flips a request.
- The ear's language and script choices are a silent failure mode on their own, separate from meaning.
- Honest limits: these are read sentences, a best case, so real-call losses are likely higher. Few speakers per language, and some languages are single-sex, so a language effect may partly be a speaker effect. Dialects differ (Brazilian Portuguese, Latin American Spanish, Egyptian Arabic). The pack has no Bengali audio, so Bengali speakers are counted as unmeasured, not zero. Census home-language counts may not be unique people. Toronto 311 is used only as a list of request types.
What's next for Lost in the Line
A pilot with the City of Mississauga 311 contact centre:
- Run the same pipeline monthly on consented real calls.
- Test any speech or translation vendor against it before switching.
- Roll out language-first menus and confirm-back in the highest-exposure neighbourhoods first.
- Extend the method to voice alerts (boil-water advisories, snow bans) before they go out.
Data sources: City of Mississauga Open Data (2021 Census; 311 call data 2017–2018) · FLEURS (Google, CC BY 4.0) · Toronto 311 2025, used only as a request-type list (Open Government Licence, Toronto). The pack also includes Speech Accent Archive (GMU, CC BY-NC-SA 4.0), MInDS-14 (PolyAI, CC BY 4.0) and Speech-MASSIVE (FBK, CC BY-NC-SA 4.0); derived transcripts from the CC BY-NC-SA datasets are released under the same licence.
AI use: ElevenLabs Scribe and Google Gemini are the systems we evaluate and part of the prototype. The pipeline code was drafted with AI assistance and reviewed by the team.
