Inspection Forecaster

Inspiration

Everyone builds chatbots on NYC open data. We asked a harder question: can an AI agent predict a restaurant's health inspection grade before the inspector arrives, and can we prove it with a backtest instead of a vibe check?

What it does

Inspection Forecaster is a Mistral Large 4 agent that looks at a NYC restaurant on the day of its inspection and predicts the grade (A/B/C), the score, and the likely violations, using only evidence dated before that day. A backtest then compares its forecasts with what actually happened and with two naive baselines.

Example (Guiz Hou Miao Jia Noodles, Flushing, 2026-06-15): last score was 25, so both baselines said A or B. The agent found a food-poisoning 311 complaint 100m away, 8 days before the visit, plus uncorrected temperature and pest violations, and forecast C. Actual: C, score 55.

How we built it

Elastic (Serverless 9.x)

• inspections index: 27,830 DOHMH inspections (one doc per restaurant + date), with a semantic_text field on violation text backed by a Mistral embeddings inference endpoint registered inside Elasticsearch.
• complaints_311 index: 100k rodent / food poisoning / food establishment complaints with geo_point locations.
• Agent tools: inspection history, geo_distance + date-range aggregation of nearby 311 complaints, zipcode baseline aggregations, and semantic search for restaurants with similar violation patterns and their next outcome.
• Backtest results are written to a forecasts index and scored live with ES|QL.

Mistral

• mistral-large-4 with function calling decides which tools to call and in what order, then returns a structured JSON forecast.
• mistral-embed powers semantic search through the Elastic inference API.

Leakage guard: every tool hard-codes date < as_of; the model cannot pick a date or a different restaurant, so it never sees the answer.

Results (30 stratified inspections per run: 10 A / 10 B / 10 C)

Challenges we ran into

v1 lost to a naive baseline: it predicted B for every actual C, because it assumed restaurants clean up after a bad inspection. The backtest exposed that, so v2 anchors on the last score and assumes no improvement without evidence. We re-scored v2 on a fresh sample (different seed) so we were not tuning to the test set. Also: Large 4 returns thinking chunks that broke structured-output parsing, so we parse the text chunks ourselves.

Accomplishments that we're proud of

An agent that is honestly evaluated. v2 catches 9 of 10 C restaurants, the ones that matter for public health, beating both baselines on that metric.

What we learned

Agents need backtests the same way models do. The most useful thing the evaluation did was show us why v1 failed.

What's next

v2 over-flags A restaurants (overall accuracy 43% vs 47% for the best baseline). Next: a confidence threshold to trade recall for precision, larger backtests, and a ranked "inspect these first" list for DOHMH. The same pattern (Elastic for geo/time/semantic retrieval + Mistral reasoning + a backtest) fits any events-plus-locations problem: work orders, insurance claims, sensor alerts.

Built With

• elastic-cloud
• elasticsearch
• es|ql
• mistral-ai
• nyc-open-data
• pandas
• python