RoadOps

A voice companion for road inspectors: log defects hands-free on the road, and let an AI agent turn every survey into repair priorities.

Comment

Inspiration

In South Africa, road inspectors still walk or drive a road, measure each defect by hand, write it on a paper form, then type it all in again at the office. The forms have no room for what the inspector actually noticed, the re-typing introduces errors, and the data is only usable days later. I wanted the data to be finished when you leave the road.

What it does

RoadOps is a voice companion for TMH9 visual condition surveys (the South African standard for road condition assessment), used from the inspector's phone. With their hands free, an inspector can:

  • Log a defect where they stand: "Log a pothole here, degree 4, extent 3." The phone's GPS goes with the request, the server works out the survey, section and kilometre, checks the values against the TMH9 rules and proposes a recommended action. The agent reads it back, and nothing is saved until the inspector says yes.
  • Photograph it: the inspector takes a photo in the app; it's uploaded and attached to the defect they just logged, and the agent says where it went ("Attached to the pothole at kilometre 35").
  • Add notes and correct mistakes: add or remove a note, change a measurement, or void an observation within 10 minutes. Each change takes exactly one yes.
  • Ask about the road: "What changed on the N1 between 2024 and 2026, and where should we spend first?" The agent chains several tools (compare the surveys, find the worst stretches, check the repair backlog) and answers in two or three sentences with the evidence. The numbers come from SQL aggregates, not from the model.
  • Pick up where they stopped: voice sessions drop and inspections take days. "I'm stopping for lunch, remind me to check the culvert" saves a bookmark; after a new session, "Let's continue" says where they were, what the previous survey found further along the section, and their reminders.

How I built it

  • RoadOps API and MCP server (.NET 9, PostgreSQL, EF Core): a clean-architecture backend for surveys, road sections and observations, with shared TMH9 validation rules, a corridor and chainage model and condition analytics. The MCP server (Streamable HTTP) has 15 task-shaped tools, from log_observation and attach_photo to compare_surveys and get_session_summary, whose answers are written to be spoken.
  • Security on the server: per-inspector API keys (only SHA-256 hashes are stored) with reader, editor and admin roles, rate limits and an audit log. The inspector's identity comes from the key, never from what the model says. Only the inspector who logged an observation can void it, and only within 10 minutes.
  • The agent on Amazon Bedrock: Alexa+ Add-ons and the MCP Toolkit are open to select partners only, so, as the rules allow, I simulated the device with the community mcp-voice-simulator and extended it in my fork. Its Bedrock brain plays the part of Alexa's LLM: Claude Haiku 4.5 through the Converse API, calling the RoadOps tools in a loop (at most 8 per turn), at temperature 0.2 so scripted flows answer the same way each time.
  • On a real phone: I added what field use needs to the simulator. The phone's GPS and photos go with each utterance as context; photos are uploaded by the simulator, so the inspector's MCP key never reaches the browser; speech recognition uses South African English; and the page works through an HTTPS tunnel (Cloudflare), protected by an access token and strict host and origin checks.
  • Safety enforced in code, not only in the prompt: a confirmation gate holds every write until the inspector's next turn, and a void is refused outright unless the inspector's own words asked for one.

Challenges I ran into

  • No public Alexa+ MCP access. I had to simulate the device and build the "Alexa brain" myself.
  • Model access on Bedrock. Newer Claude models were listed for my account, but failed with an AWS Marketplace AccessDenied after one or two calls. I made the model configurable, added a smoke-test script, and settled on Claude Haiku 4.5.
  • Phones need HTTPS for the microphone, GPS and camera, so a local simulator needed a tunnel, an access token and host/origin checks that still block cross-site requests.
  • Speech recognition outdoors heard "pothole" as "portal" and swallowed "degree". The agent now asks one short question instead of guessing, and the read-back is the safety net.
  • A "yes" that confirmed the wrong thing. In a rehearsal, "log a pothole here" next to an earlier pothole made the model propose voiding the earlier one, and the "yes" meant for the log went to the void (the server's 10-minute rule stopped it). Now the app refuses any void the inspector didn't ask for.

I kept all of these, with suggestions for AWS and Amazon, in a friction log.

Accomplishments that I'm proud of

  • Tested on a real Android phone, outdoors and indoors: a defect logged by voice, read back, confirmed and photographed without a paper form or re-typing at the office. (The test data covers the N1 and other national routes, so the positions in those tests were typed or given as a kilometre.)
  • Memory between sessions, so a dropped connection or a lunch break doesn't lose the inspector's place.
  • Safety rules enforced in code on both sides, each found or proven by testing on the phone (manual test results).

What I learned

  • Designing tools for a model is not the same as designing a REST API. A few task-shaped tools ("log this defect here") worked better than create, read, update and delete endpoints, and every answer had to be short enough to say aloud.
  • A confirmation step is only as safe as what the person thinks they are confirming. My first gate checked that a "yes" came on the next turn, but not that the "yes" was meant for that action, and the void in a rehearsal showed the difference.
  • Testing on a real phone found what unit tests couldn't: Android's approximate location, speech recognition hearing "portal" for "pothole", and a model that sometimes read back from its own words instead of the server's.
  • The TMH9 rules fit naturally on the server: degree and extent ranges and distress names are checked there, and the recommended actions are computed there, so the model only has to explain them.

What's next for RoadOps

  • The full assessment cycle: plan from past surveys, re-inspect with guidance to known defects (marking each fixed, unchanged or worse), compare the surveys, and recommend treatments such as moving from patching to resurfacing.
  • Offline capture and sync for rural roads with weak signal.
  • OAuth sign-in in place of API keys.
  • A real Alexa+ Add-on once MCP Toolkit access opens up.

Built With

Share this project:

Updates

Submission history