Mamdani

Inspiration

NYC Mayor Zohran Mamdani went viral for meeting residents where they are: on the street and on camera, asking what's broken. Most people never get that. Reporting a pothole in Ottawa today means finding the right 311 form, guessing a category, describing it in words, and then hearing nothing back. On the other side, city staff get a pile of vague, duplicated tickets with no photo and no sense of what matters most.

And it worked because it was public. Mamdani's videos weren't just him listening; they were everyone watching him listen. When a problem is visible, neighbours pile on, and the people in charge have a reason to answer. That's the cycle we wanted to build: a resident raises something, the community backs it up, the city responds in the open, and everyone sees it get fixed.

We wanted everyone to have a Mamdani in their pocket who shows up, looks at the problem with you, and makes sure City Hall actually sees it.

What it does

For residents: open the app, point your phone at the problem and just talk. Mamdani, a 3D animated inspector, sees what your camera sees and hears what you say. He asks what he needs to and takes the evidence photo himself. No forms, no categories to guess. Then he suits up for the job (hard hat, flag, cone) and tells you what happens next. You can follow your report from "waiting" to "crew assigned" to "fixed", and anything he tells you about it is checked against the city's record first.

Behind the scenes: Gemini turns the conversation and the photo into a work order a crew can act on:

what the problem is and where it is in the photo how severe it is and how dangerous whether it blocks a wheelchair which department owns it

Faces and licence plates are blurred before anything is stored. When ten people report the same pothole from different angles, it becomes one work order with ten confirmations, matched by location and by what the photos actually show. Every confirmation raises its priority.

For everyone: Mamdani Social. The app's second tab is a public feed where every report becomes a post: the resident's photo, their own words, and Gemini's summary. Neighbours tap "Me too" and comment to say it affects them too. Those "me too"s are the same confirmations that raise the issue's priority on the city dashboard, so community support directly moves it up the crew's queue. As the city works on it, its replies appear right under the post ("Crew assigned · Roads Services", "Fixed"). It's the Mamdani cycle, built in: complaints come together in public, the city answers in public, and everyone sees the result.

For the city: Mamdani Command, a dashboard for government supervisors, inspectors and dispatchers, behind Auth0 sign-in:

Explainable priorities: every issue is ranked by a score you can open up (severity, safety, accessibility, number of reports, time waiting), with a clock against its service target. Live 3D map: every open issue, a heat layer of where residents report, and a 7-day replay. Crew routing: pick the jobs and it builds the shortest run from the Public Works yard on real roads, ready to send to the crew. Gemini work plans: crew and hours, a cost worked out line by line from planning rates, materials, traffic control, the sourced risks of waiting, and a plain-language update for the residents who reported it. A daily brief: Gemini reads the whole record, checks the weather and conditions with Google Search, and writes the morning memo on what to fix first and why. Ask Mamdani: he lives in the corner of the dashboard. Staff ask him anything ("which accessibility barriers are overdue in Centretown?") and he answers from the live database, cites sources when he uses the web, highlights what he's talking about on the map, and proposes changes a person confirms.

New reports appear on the city dashboard and in the public feed within seconds, so residents, neighbours and the city are all looking at the same record.

How we built it

Mobile (React Native, Expo development build): streams camera frames and microphone audio to the Gemini Live API (gemini-3.8-live, Orus voice). Mamdani runs the conversation and calls a report_issue tool when he's ready to take the photo. Each report gets an idempotency key, so a retried upload never files twice. The app has two tabs: Report (Mamdani and the camera) and Feed (Mamdani Social), a native React Native feed built on the same API and database as the dashboard. A post, its "me too"s and the city's replies are the same records the city works from.

Mamdani himself was generated from a single reference image and auto-rigged with Tripo. We baked the model into a compact format so the same rigged character runs in three.js on both the phone and the web, with procedural walking, gestures, blinking and a mouth that follows his voice. The waiting music was generated with Lyria.

AI pipeline (Next.js on Vercel, Vertex AI):

gemini-3.8-flash analyzes the photo, optional video and conversation. It returns category, severity, safety risk, accessibility impact, bounding box and confidence, and also chooses Mamdani's outfit, animation and line. gemini-3.5-flash-lite screens photos, finds faces and plates to blur, and double-checks duplicate matches visually. gemini-embedding-2 turns each evidence photo into a 768-dimensional vector. The same vectors let staff search the evidence by describing it in words. Vertex AI Check Grounding verifies what Mamdani tells residents against the saved work order, with Flash-Lite as a fallback judge. On the dashboard, gemini-3.8-flash powers a tool-calling agent, the daily brief and the work plans. Google Search grounding brings in outside facts with sources, and Gemini TTS gives dashboard Mamdani the same Orus voice as on the phone.

Tiger Data (TimescaleDB + pgvector) is the city's memory:

Every resident report lands in a hypertable. A continuous aggregate powers the hourly activity charts. Duplicate matching runs as a pgvectorscale DiskANN similarity search on photo embeddings, combined with a location filter. Today's brief is written once and stored there, so every visit reads the same memo.

Dashboard: React + TypeScript + Vite, Mapbox Standard (3D, day/night), GSAP animations, Auth0 sign-in, deployed on Vercel.

Challenges we ran into One pothole, ten reports: people photograph the same problem from different angles, at different times, from slightly different spots. GPS alone merges things it shouldn't, and photos alone miss things. We combined a location filter, photo-embedding similarity and a Flash-Lite visual check. Keeping the AI honest: a friendly character talking to the public can't make things up about a city work order. Every answer about a report is checked against the record, and unsupported claims are rewritten from the facts alone. Work-plan costs are calculated from stated rates, and dollar claims only appear when a cited source backs them. Live latency: streaming audio and video to Gemini Live through our backend, with guardrails, while animating a 3D character on a phone, meant fighting for every frame and every millisecond of audio. Making it feel human: timing Mamdani's gestures, glances and mouth to his speech took far more iteration than the AI plumbing. Accomplishments that we're proud of The full loop works end to end: a report made by talking to a phone shows up on the city dashboard seconds later, deduplicated, prioritized and on the map, and status changes flow back to the resident. Three views of one record: the resident's phone, the public feed and the city's dashboard all read the same Tiger Data record, so a single report instantly becomes a work order, a public post and a point on the map. One character across the whole product: the same rigged 3D Mamdani greets residents on the phone and answers city staff on the dashboard, in the same voice. Every AI output is explainable or verifiable: priority scores break down into their parts, costs show their working, web answers cite sources, answers to residents are grounding-checked, and the assistant proposes changes instead of making them. What we learned Multimodal AI is strongest when it stays grounded: the database is the source of truth, and Gemini reads it, reasons over it and explains it. Tool calling turns a chatbot into a colleague. Once Mamdani could query the record, draw on the map and plan routes, staff stopped asking "what can it do?" and started asking about their streets. Accountability is a feature. Making reports visible to neighbours turns individual complaints into shared ones, and gives the city a reason to respond, not just a ticket to close. Good civic tech is mostly about closing the loop: the resident feeling heard matters as much as the report being accurate. What's next Real accounts and verified City staff (and the mayor) replying in the feed straight from Mamdani Command, with before/after photos when a fix is done. Community voting that feeds the priority score directly, with moderation to keep the feed civil. Push notifications when a crew is assigned and when the fix is done. Connecting to real 311 and work-order systems, and to open data like road-condition surveys, new developments and social programs, for long-term quality-of-life forecasting. Equity reporting for councillors: which neighbourhoods wait longest and where accessibility barriers cluster. ShieldGemma content screening, and support for more languages.

Built with: Gemini Live API, Gemini 3.8 Flash, Gemini 3.5 Flash-Lite, Gemini Embedding 2, Gemini TTS, Vertex AI, Vertex AI Check Grounding, Google Search grounding, Lyria, Tiger Data (TimescaleDB, pgvector, pgvectorscale), Auth0, Tripo, three.js, React Native, Expo, React, TypeScript, Vite, Next.js, Vercel, Mapbox, GSAP

Built With

• auth0
• claude
• codex
• gemini
• nextjs
• postgresql
• python
• react
• tigerdata
• tripo
• vertexai