311Relay

311Relay: text a Telegram bot what’s wrong in NYC. It finds the right agency, drafts the complaint, and after you approve, files it on the official form and returns the real reference number.

Comment

Inspiration

NYC311 lists hundreds of complaint types, each with its own subcategories and its own form, often run by a different agency. Most people who see a problem never report it, because finding the right form takes longer than the problem seems worth. We wanted reporting to be as easy as texting a friend: say what's wrong, and the right report gets filed.

What it does

311Relay is a Telegram agent. You send a photo, a sentence or a voice note, for example "A train stuck at 14 St for 20 minutes, no announcements."

  1. It classifies the issue and works out which agency and form it belongs to.
  2. It asks only for what that form needs and you haven't said yet (for a subway delay: line, direction, station, time).
  3. It checks live MTA service alerts and quotes any matching alert on the draft.
  4. It shows you a draft card with exactly where it will be filed and the exact text it will submit.
  5. Nothing is sent until you tap Approve.
  6. A disposable Daytona sandbox launches an isolated worker that drives a real Anchor cloud browser through the official form.
  7. You get the official reference number (only if the confirmation page shows one), the screenshots and a video recording of the whole filing.

If a form needs a human (a CAPTCHA or a login), 311Relay fills what it can and hands you the live browser session to finish. It never gets around the CAPTCHA.

Today, subway delays file end to end on the official MTA Customer Feedback form. The category registry and the worker adapters are pluggable, so each new category is one registry entry plus one adapter.

How we built it

  • Bot and orchestrator: TypeScript, grammY for Telegram, and a deterministic state machine for each complaint (draft → ready for approval → approved → filing → filed / handed off / submission unknown). The LLM classifies and plans. Code owns every state change.
  • Reasoning: OpenRouter with strict JSON-schema structured output, plus vision for photos.
  • Execution: each filing attempt gets its own Daytona sandbox running a filing worker that's locked to an allowlist of domains.
  • Browser: Anchor Browser, driven by Playwright over CDP from inside the sandbox, with screenshots at every step and the session recording sent back to Telegram.
  • Memory: SurrealDB, holding a graph of residents, complaints, locations, filing attempts and official requests.
  • Observability: Arize Phoenix with OpenTelemetry: one session per chat, a trace for every model call and filing step, and an evaluation suite covering missing locations, duplicates, ambiguous photos, cancellations, CAPTCHA hand-offs, losing the connection after submit, emergencies and out-of-scope requests.
  • Dashboard: a read-only Next.js console showing complaints, attempts, evidence and recordings.

Challenges we ran into

  • CAPTCHAs everywhere. The NYC DOT pothole form and the NYC311 forms we probed all load reCAPTCHA. We chose to hand off rather than get around it, so we went looking for official forms that a bot can legitimately fill. The MTA's feedback wizard has no CAPTCHA and no login on any of its 11 steps.
  • Mapping a real wizard. The MTA form is a Salesforce + Skuid app. Its choice buttons have no visible text, only accessibility labels, Continue stays disabled until you pick something, and station lists load late. We mapped every step with a headless browser, stopping one click before Submit, and wrote the adapter from that map.
  • "Towards Manhattan" isn't an option. On our first live run, the A train's two directions were "Uptown (Inwood-207 St)" and "Downtown & Queens." The worker correctly stopped instead of guessing. We fixed it by matching the direction against the MTA's own options for each line before approval, and asking with two buttons when it's still ambiguous.
  • Never inventing a receipt. We couldn't see the confirmation page without actually submitting, so the receipt parser only accepts a number that appears on the page. Anything else becomes SUBMISSION_UNKNOWN with evidence, and the worker never retries.

Accomplishments that we're proud of

  • Every filing goes to the official form. There's no fake portal or demo site in between.
  • A full subway-delay dry run on Daytona + Anchor in about 53 seconds: all 11 steps filled, stopped one click before submit, with screenshots and a video.
  • Safety rules enforced in code: only a resident's tap approves a draft, the worker submits at most once, it never fills contact details you didn't give, and a real submission needs both your approval and an explicit server flag.

What we learned

  • Real government forms are messy and change without notice. The agent has to be honest when it's unsure, not confident.
  • Letting the LLM classify and plan while deterministic code owns the outcomes made the system much easier to trust and to test.
  • Observability isn't only for debugging. Traces, screenshots and recordings are the evidence a resident can check.

What's next for 311Relay

  • Agent memory: "Someone reported this an hour ago. Do you want to add yours?" across all residents.
  • Category catalog: crawl the whole NYC311 directory and send you the official link instantly for anything we can't file yet.
  • Learned required fields: when a form asks for something new, record it in memory and ask for it up front next time.
  • Self-repair: use Phoenix traces, Anchor recordings and screenshots to propose adapter fixes when a form changes, with a human approving each fix.
  • More forms with no CAPTCHA: other MTA branches (buses, elevators, LIRR) and the FTC robocall report.

Built With

Share this project:

Updates

Submission history