Quote Runner

An agent that turns messy fabrication enquiries into priced quotes, where the model never names a price and only deliverable, profitable quotes pass

Comment

Inspiration

We run a two-person fabrication lab in Qatar. FDM printing, SLA, CNC routing, milling, turning, laser cutting. The machines are the easy part. The bottleneck is everything that happens before a machine turns on.

A typical enquiry arrives as a WhatsApp photo of a hand-sketched bracket, or an email with a STEP file and the sentence "how much and how fast?" Turning that into a quote means reading the geometry, choosing a process, estimating machine time, checking whether the stock is on the shelf, checking whether the part even fits inside the machine, applying a margin, and formatting the result.

That is 30 to 45 minutes per enquiry, several times a week, usually at night. It is also inconsistent. The same part quoted twice a month apart can come out 15% apart depending on how tired the person quoting it was.

An obvious automation target. We built the obvious thing, and then hit the question that turned out to be the actual project.

How do you know an automatically generated quote is any good?

Every cheap answer is a trap. "The customer accepted it" rewards an agent that quotes everything at cost. "It looks professional" rewards formatting. "The model was confident" rewards a model that is always confident. Optimise a quoting agent against a proxy and you do not get a better quoting agent. You get an agent that has found the cheapest route to satisfying your proxy. In this domain that failure has a specific shape: it prices the shop into bankruptcy, politely, at scale, while every dashboard reads green.

Quoting is an unusually clean place to demonstrate this, because the failure is denominated in currency and you can watch it happen against real machine constraints.

So Quote Runner is a working quoting agent, and it is also a rig for measuring what happens when you point that agent at the wrong objective — and, as it turned out, for measuring whether you can detect anything at all.

What it does

Quote Runner takes an unstructured enquiry and returns a priced quotation or a reasoned refusal.

  1. Ingest the enquiry as free text. Attachment filenames are passed through as context; the agent does not read file contents.
  2. Extract the job: part, quantity, material, dimensions, tolerance hints, deadline.
  3. Plan the work against the lab's actual machine list — fdm_01, fdm_02, sla_01, laser_01, router_01, mill_01, lathe_01 — selecting a process, an operation sequence, and estimated machine minutes.
  4. Price it deterministically. The model never produces a number.
  5. Present the quote in a browser with its price provenance stated on the face of it, and the agent's real tool-call sequence shown underneath.

It refuses as readily as it quotes. A part longer than the machine's travel, a material out of stock past the deadline, or an enquiry with no recoverable dimensions comes back as an escalation with the specific blocker named — which axis, by how many millimetres, or which fact is missing.

The two judges

The honest judge is conjunctive. A quote passes only if it is both deliverable and profitable, and both halves must hold:

pass(q) = (stock available AND geometry inside envelope AND lead time survives queue)
          AND (margin after material + machine + labour >= floor)

The gameable judge scores one axis: would the client accept this price. No stock check, no envelope check, no queue check, no margin floor.

The gameable judge is broken on purpose and we are not improving it. Its brokenness is the demonstration.

The numbers

Same agent, same run, same 17 development cases, scored under both judges in a single pass:

Judge Pass rate
Honest — deliverable AND profitable 70.6%
Gameable — would the client accept the price 100.0%

Five quotes in that gap lose the shop money or describe work it cannot do, and the second judge cannot see one of them.

The oracle — ground truth played straight into the engine — scores 100% on all 25 cases, which is what tells us the cases are internally consistent rather than unsolvable.

How we built it

Stack

  • Gemini 3.5 Flash via Vertex AI for enquiry parsing and job planning. Flash over Pro deliberately: the reasoning steps are narrow and frequent, so latency and cost mattered more than depth.
  • Google ADK for agent orchestration and the tool surface.
  • Cloud Run hosting the service and the customer-facing web UI, one container serving both. Scale-to-zero kept the entire build inside a few dollars of credit.
  • Vanilla HTML, CSS and JavaScript for the UI. No React, no npm, no bundler, no build step. A hackathon deadline is the wrong moment to be debugging a toolchain.

The pricing wall

The model never names a price. Not once, not anywhere, not even as a suggestion the code later overrides.

Every figure a customer sees originates from price_job() in costing/, a deterministic function over material, machine time and labour. The rate card it reads is not invented: derive_rate_card.py generates it from the workshop's own equipment costs, and every machine rate is traceable to a replacement value, a depreciation schedule and a division's staffing.

The separation is enforced rather than intended. A priced_without_tool flag records any attempt by the model to arrive at a number on its own, and a server-side gate refuses to release a quote unless the price clears the engine floor and that flag is false. A model-authored price does not reach the UI; it produces its own error panel naming what went wrong.

Provenance sits on the face of the quote, not three clicks down. Directly beneath the headline price is a line stating the engine floor and the percentage above it. Hiding provenance in an expandable section is the same thing as hiding it.

The reasoning is simple. A hallucinated dimension is recoverable, because the customer corrects you. A hallucinated price is not, because the customer accepts it.

The optimisation loop

We wired GEPA in to close the honest-judge gap. It rewrites agent/prompt.py against failed evaluations, and costing/ is out of scope permanently — a prompt optimiser with write access to the pricing engine will eventually discover that the fastest route to a passing margin is editing the margin floor.

The eval set is 25 hand-authored cases with deliberately seeded traps: geometry outside the machine envelope, materials out of stock, dimensions simply missing. Eight are held out and scored exactly once, after the loop finishes; the optimiser raises if it is ever handed one. We have no historical accept/reject data, so the cases are constructed rather than sampled, and we say so rather than dressing them up as a backlog replay.

Challenges we ran into

The optimiser accepted a prompt it had already rejected twice

This is the finding we did not expect and did not go looking for.

GEPA ran six generations against the honest judge and accepted exactly one rewrite — generation 3, at 82.35%. Then we hashed the prompt files.

gen  sha256[:10]   dev_honest  passed  decision
0    6b98b351a9       0.7647    13/17  seed
1    2990b36c00       0.7647    13/17  rejected
2    2990b36c00       0.7059    12/17  rejected
3    2990b36c00       0.8235    14/17  ACCEPTED
4    7ab0c2b2d2       0.6471    11/17  rejected
5    0fe7ffd240       0.5882    10/17  rejected
6    0fe7ffd240       0.7059    12/17  rejected

Generations 1, 2 and 3 are byte-identical. Generation 3 did not beat generations 1 and 2 — it is generations 1 and 2. The same text scored 76.47%, then 70.59%, then 82.35%. Rejected twice, accepted on the third sample. The end-of-run re-score evaluates that same text a fourth time and returns 64.71%.

Four evaluations of bytes that never changed: 13, 12, 14 and 11 out of 17. Generations 5 and 6 reproduce the pattern independently.

Nothing malfunctioned. The coach proposes from the current best instruction, so while best stayed at generation 0 it received the same parent and the same failure set three times and returned the same text three times. The acceptance rule is strictly-greater-than against current best, and that has no defence against an identical candidate eventually sampling above the bar.

We verified this by hash rather than by reading the prompts, because reading them is how you talk yourself into seeing an improvement that is not there.

The measurement is not sensitive enough to detect what it was pointed at

The dev set is 17 cases, so one case is worth 5.88 percentage points. GEPA's entire search spanned four cases. The variance on an unchanged prompt is three.

We corroborated this deliberately: scoring the seed prompt five separate times returned 70.6%, 70.6%, 64.7%, 70.6%, 70.6%. Combined with every other scoring of that unchanged prompt across both runs, the range is 58.8% to 76.5%.

A great many reported agent improvements at this scale are measuring sampling variance. We built the loop, ran it properly, and the honest answer is that the instrument could not resolve what it was measuring. Reporting that is more useful than a chart with an upward slope on it.

A broken metric does not mislead an optimiser. It switches it off.

Running the same loop against the gameable judge produced zero acceptances in six generations — and the reason is more interesting than the result.

LlmCoach.propose opens with if not failures: return instruction. The gameable judge passes 17 of 17 at generation zero, so the failure list is empty, so the coach returns before constructing a prompt. Zero coach model calls across the entire run. We confirmed the six prompt files are byte-identical to the seed by hash.

The failures were there the whole time — five to seven per generation, recorded in the same result files under the honest verdict. The judge doing the selecting could not see them. Six generations of reported success, and the optimiser never once ran.

Our own reporting script had its conclusion hardcoded

show_divergence.py printed a fixed sentence announcing that the proxy-optimised prompt performed worse on held-out cases. It was true when written. After we corrected the eval set and re-ran, both arms scored 50.0% held-out — the gap is zero — and the script would have announced a divergence anyway, on screen, in our demo video.

We built a project about metrics that report success while the thing underneath fails, and we put one in our own tooling. It now derives that sentence from the data, and the three branches were tested against synthetic fixtures so no branch can assert a conclusion the numbers do not support.

We audited our own eval set and found three problems

Late in the build we checked all 25 cases against the rate card rather than trusting them:

  • One PLA mass was physically impossible — 5.0 g for a part whose solid volume weighs 0.54 g. It turned out to be the solid mass of the part with one dimension left in the client's mistaken centimetre reading, which is the exact trap that case was written to test. Fixed to 0.40 g, derived from geometry.
  • Two brass jobs the client described as turned were assigned to a 3-axis mill. The rate card had no lathe. We added one, derived its rate the same way as every other machine, and split the second case into turning plus cross-drilling.
  • Adding the lathe surfaced a queue bug: the engine summed queue hours across machines, so a two-machine job waited twice. Machine backlogs drain in parallel. Corrected to a single job-level delay, with a regression test, because none of the 16 existing tests covered multi-machine queue aggregation.

Every eval figure in this write-up was re-derived after those corrections.

Vertex rejected our tool schema

Typed nested lists in the tool definition generated $defs and $ref in the JSON schema, which Vertex refuses. The fix was to pass operations as a JSON string and parse it on the other side. Not elegant, but the alternative was flattening the operation model into something that no longer described the work.

A verifying agent conformed to results we showed it

Partway through, we noticed that handing a verifying agent the prior expected results caused it to agree with them rather than check independently. Same failure as the gameable judge, one level up, inside our own tooling. Fixed by supplying prior results as reference material with an explicit instruction not to conform, and later by withholding them entirely and comparing afterwards.

Accomplishments that we're proud of

  • A pricing boundary enforced in code, not prompt text. The model cannot price a job. Not "is instructed not to" — cannot, behind a gate that fails closed.
  • A measured noise floor. Thirteen scorings of one unchanged prompt, and a re-roll caught by hash comparison. Most projects at this scale report an improvement. We can show why ours would not have been detectable.
  • Null results reported as null results. Zero coach calls, byte-identical prompts, and a held-out divergence that did not reproduce, all in the write-up with the verification, rather than replaced by a story about how much we learned.
  • A rate card derived from real equipment costs. Every machine rate traces to a replacement value and a depreciation schedule from the workshop's own accounts.
  • An interface that does not lie about what the agent did. The activity list shows real tool calls with real durations. No invented intermediate steps, no artificial delays.
  • Most claims verifiable without credentials. test_costing.py, evals.harness --validate and show_divergence.py run from a fresh clone on the standard library alone — no install, no Google Cloud account.

What we learned

  • The metric is the product. We spent more time on how to evaluate a quote than on how to generate one, and that was the correct allocation. A generation pipeline with a bad objective is a machine for producing confident errors at speed.
  • Conjunctive metrics resist gaming in a way weighted ones do not. Deliverable AND profitable cannot be satisfied by winning hard on one term. A weighted score of the same components can be, and would have been.
  • Verify by hash, not by reading. The re-roll finding was invisible to anyone reading the prompts and obvious the moment we compared checksums.
  • Deterministic and probabilistic components need a wall, not a convention. Everything that must be correct belongs on the deterministic side, and the wall needs a gate that fails closed.
  • The failure modes are fractal. The gameable judge optimises for the wrong proxy. Our reporting script hardcoded its conclusion. A verifying agent conformed to results we handed it. The temptation to render satisfying fake progress steps in the UI is the same instinct. One error, four scales — and we only caught the last three because the first one taught us to look.

What's next for Quote Runner

  • Grow the eval set past the noise floor. Roughly 200 cases before any optimisation result is worth reporting. This is the blocking dependency for everything else.
  • Score every candidate more than once. The re-roll happened because a single sample per generation cannot distinguish a better prompt from a luckier roll of an unchanged one. Repeated sampling with a variance-aware acceptance rule is the fix, and it costs quota rather than cleverness.
  • Resolve the two open eval-set questions. Sheet-goods nesting yield is assumed at 20–30% across eight cases and has never been checked against a real nest. Whether material_grams_per_unit means finished part or consumed stock is documented but should be settled.
  • Track outcomes. Accepted and rejected quotes fed back so margin bands calibrate against real win rates rather than our assumptions.
  • Multi-tenant. Every small fabrication shop has this exact problem and none of them have a software team.

Disclosure

Built during the submission period, starting 23 August 2026. Claude Code was the primary build executor throughout; the architecture, the eval design, the judge definitions and every correction in this write-up were directed and reviewed by the author. No pre-existing project code was reused.

Built With

Share this project:

Updates

Submission history