Advil

• 

  Results page from the final run

• 

  Persona UI

• 

  Persona UI NOT APPROVED

• 

  Persona UI Reasoning

Inspiration

Support agents send a lot of emails, and one wrong number can cost real money. Tell a customer their loyalty discount is 30% when the policy says 15%, and the company either eats the difference or breaks its word. AI drafting tools make replies faster, but they also make it easy for a wrong fact to go out sounding confident.

We wanted a gate that sits right before send_email and asks one question: does this email match what the company actually offers?

Ezio gives useful signals: tone, bias, agreement, and a fact-check for public claims like laws. It can't see a company's own price sheet, though. In the practice set, most of the bad emails came back GREEN with accuracy not scored. A GREEN score only tells you what Ezio could see, so the gate needed its own check.

What it does

For every draft reply, the wrapper:

1. Calls Ezio and reads its overall tier, each dimension score, and its per-claim fact-check.
2. Checks each sentence against the company's price_sheet.csv and policies.json:
   • plan prices (monthly, annual, after the loyalty discount)
   • discount amounts, which plans they apply to, and stacking
   • loyalty eligibility, using the customer's plan and how long they've been a customer
   • refund windows, first-response times, free trial length and plans
   • promises an agent can't make: features, release dates, exceptions, guarantees
3. Marks every claim verified, contradicted, or unverified, along with the file it was checked against.
4. Returns APPROVE, HOLD_FOR_REVIEW, or BLOCK with a reason and evidence, and writes it to an audit log. The email is only sent on APPROVE.

The persona page is built for the support lead. You enter the draft and the customer's plan and tenure, then see a green, amber, or red result, a plain line saying whether send_email ran, and each claim with its evidence.

How we built it

• Python standard library only. The wrapper serves POST /decide and calls the Ezio mock over HTTP.
• One small check per business rule. Every value comes from the data files and is never typed into the code, so if the price sheet changes, the checks follow it.
• Number parsing for how people actually write: "nineteen dollars", "$99/mo", "within fourteen days", "the first year". Each number is matched to the nearest plan named in the sentence, or to the customer's plan if none is named. That's how "Business is $99/mo with a first response within 24 hours" gets caught: the price is right, but the response time belongs to a different plan.
• Thresholds from the data. Normal friendly emails scored 90+ on tone, and the over-the-top example scored 26. We hold below 60, which leaves room on both sides.
• Fail closed. If Ezio is down, rate-limited past its retries, or sends a broken response, the email is held for a person.
• The persona page is one HTML file with no outside libraries, so it works offline.

We used Claude Code to draft the check logic, the policy justifications, and the persona page. We ran the practice set after each change and wrote 20 extra test sentences of our own, to make sure the rules worked on new wording and weren't just fitting the practice set.

Challenges

• Not over-blocking. Blocking everything fails as badly as approving everything. "I'll send you an update by end of day" is fine, and "I'll make sure engineering ships that feature next month" is not. Telling those apart took care.
• Stacked discounts. A stacked discount can show up as a simple sum (15% + 20% = 35%) or compounded:

$$1 - (1 - 0.15)(1 - 0.20) = 0.32$$

We check for both.

• "Business" the plan vs. "your business." We only treat it as the plan when it's capitalized or followed by "plan."
• What a RED tier should mean. We decided a BLOCK needs a specific wrong fact. A low overall score alone sends the email to a manager.
• A miss we found during judging. A surprise input, "I promise we'll have the new dashboard ready for you by Friday," was approved when it should have been held. Our promise check looks for words like "feature," "ship," and "release," and this sentence used none of them. The fix is to treat "I promise" as a hold on its own, and to flag anything promised to be ready by a deadline.

What we learned

• Read each signal on its own. An email can be GREEN overall with a tone score of 26.
• Measure both directions:

$$\text{catch rate} = \frac{\text{bad emails not approved}}{\text{all bad emails}} \qquad \text{false-block rate} = \frac{\text{good emails not approved}}{\text{all good emails}}$$

Pushing one up without watching the other is easy and useless.

• Evidence matters as much as the decision. A manager can act on "the price sheet says $99, the email says $79." A bare "BLOCK" gives them nothing to act on.

Results

• Final set: 50.0 / 50 (6/6 bad emails caught, 0/6 good emails blocked, 12/12 exact, 6/6 stretch, 6/6 failure drills)
• Practice set: 16/16 exact, failure drills 15/15

What's next

• Fix the promise check described above.
• Show Ezio's sources as a clean publisher name and link.
• Add a review step on the page where a manager approves or rejects a held email with a logged reason, plus a view of the audit trail.

Built With

• ezio
• python