FieldPilot

The LLM never builds the route. Gemini reads what rules cannot - voice notes, photos, free text - and prices each job; OR-Tools optimizes. Every claim measured against hidden ground truth.

Comment

Inspiration

I spent 20 years implementing Microsoft Dynamics 365 Field Service. Every dispatcher I met made the same complaint: the scheduling engine reads structured fields, but the thing that decides whether a job actually matters lives in free text — the note that says a frail resident has no backup heating, or the one that says the boiler already restarted on its own. Rules engines cannot read those. Language models can. But nobody wants an LLM inventing routes.

So FieldPilot is built on one sentence: the language model does not build the route — it writes the cost function that the solver optimises.

What it does

FieldPilot dispatches a simulated HVAC/gas fleet in Buenos Aires, end to end:

  • Intake (Gemini, multimodal): a typed message, a voice note, or an equipment photo becomes a structured work order, geocoded to a ROOFTOP-precision point. No real geocode, no dispatch.
  • Triage (Gemini): the whole 48-order backlog is scored in one call. Each job gets one bounded integer — the cost of leaving it undone today.
  • Planning (OR-Tools VRPTW): minimises travel plus the penalties of whatever it cannot fit. Solution-limited, so every plan is reproducible on any machine.
  • Monitoring (Gemini): breakdowns, overruns and emergencies stream in; the monitor answers one verb — REPLAN or ABSORB — and the solver redraws only what has not started.
  • Comms (Gemini): customer notifications drafted from computed facts and verified before sending — every number must exist in the facts, commitment language is refused, and a rejected draft is replaced by a deterministic template.
  • Escalation queue (no model, by design): unserved safety calls, out-of-area geocodes, twice-missed customers. A queue a model could talk itself out of raising would not be a safety net.

Every model output crosses one boundary as a named, bounded value with a deterministic fallback. The model was measured changing its severity opinion across four identical production requests — the dispatch was identical all four times, with each override logged.

How it is measured (the part I care most about)

Every claim runs against a hidden ground truth: the simulator assigns each job a true importance no triage method ever sees, and an oracle ceiling shows how much margin existed at all. Results, over 18 seeds:

  • The full system beats its per-seed baseline on 16 of 18 days (+14.6 pts; a coin does that p<0.001). Propagated lateness falls 368→30 min; emergency completion doubles (16/42→34/42).
  • Gemini captures 91% of the margin hidden in free-text notes; a good-faith keyword scanner captures 40%.
  • Negative results are published too: per-technician duration learning measured +3.6 ±5.2 against an oracle ceiling of +3.0 — near zero, and the README says so. An early "triage and monitoring are superadditive" claim died at 6 seeds and stayed dead at 18; the retraction is in the README.

How I built it

Python 3.11. Google ADK (LlmAgent + InMemoryRunner) over Gemini 3.5 Flash on Vertex AI; Google Cloud Run (deployed, verified end-to-end by the deploy script itself), Cloud Storage (shared geocode cache), Secret Manager (Maps key), Cloud Build, and the Geocoding API. OR-Tools for the VRPTW. FastAPI for the service layer. 240 tests.

Challenges

Measurement kept humbling me. A 4-of-4 severity instability "finding" did not replicate at 10 trials; a +21.5 superadditivity died at wider samples; the solver was quietly reading the simulator's ground-truth technician speeds until I removed the leak and re-measured everything. Cloud Run reserved /healthz and 404ed it before my container ever saw the request.

What I learned

Split the work by what each side is reliable at: the model does what needs language (it returned the same incident type 10/10 times); the taxonomy and solver own what needs to be identical twice (severity, skills, routes). And measure against ground truth the evaluated system cannot see — half of what I "knew" during the week did not survive that.

Third-party and pre-existing code

Google OR-Tools, Google ADK, google-genai, google-cloud-storage, FastAPI, Pydantic, pytest (all via pip). All FieldPilot application code was written new during the submission period.

Built With

Share this project:

Updates