Inspiration
A support agent drafting customer emails with AI is one bad sentence away from promising a discount the company doesn't offer. Trust scores help, but a high score is not proof the facts were checked: in our very first test, the mock Ezio rated "Your 30% loyalty discount applies to the Pro plan" GREEN 94, while the real loyalty discount is 15%. We wanted a gate where the model that drafts an email is never the one that decides whether it goes out.
What it does
Lumen Cloud Reply Check sits between a drafted customer email and send_email. For every draft it:
- checks every price, discount, refund window, response time and trial against the company's price sheet and policies, using the customer's plan and tenure (written-out numbers, several facts per sentence, weeks and months included);
- uses Ezio for what Ezio can see: contradicted public claims (e.g. GDPR) block, CRITICAL findings and low tone/bias/agreement go to review;
- holds promises of features, delivery dates or exceptions, plus hype and flattery, for a manager;
- returns APPROVE, HOLD_FOR_REVIEW or BLOCK with a plain-language reason and evidence, and runs
send_emailonly on APPROVE.
The persona page lets an agent check a draft or simulate a test case, shows the decision, reason, evidence and Ezio's signals, and whether the email was sent. Held emails get a manager-review step that records a reason and never sends. An audit-trail view shows every decision.
How we built it
Python standard library on the provided starter. checks.py parses drafts and compares them with price_sheet.csv and policies.json; wrapper.py combines those findings with Ezio's signals under a written policy (DECISION_POLICY.md, one sentence of justification per threshold); audit.py writes a hash-chained audit log, so an edited or deleted record is detected (the page shows "hash chain verified"); the page is plain HTML/JS calling only POST /decide.
Challenges we ran into
- Score traps: high Ezio scores on factually wrong emails, so facts had to come from company data, not the blended score.
- Failing closed without over-blocking: when Ezio is down, slow or malformed, our own checks still run; a provable error is blocked, anything else waits for a human, and nothing is sent unverified.
- Near-misses: "within one business day" isn't the Business plan; "no credit card required" isn't a card requirement.
Accomplishments that we're proud of
- Final held-out run: core 12/12 exact, 6/6 bad caught, 0/6 false blocks, stretch 6/6, failure drills 15/15 (50/50 measured).
What we learned
A trust score is a signal, not a decision. The useful work is deciding which signal is authoritative for which kind of claim: company data for prices and policy, Ezio for public facts and tone, and a human for anything the agent isn't authorized to promise.
What's next for SCC
Learn new prices and policies automatically from the company's systems, add per-customer context (contract terms, open tickets), and route held emails into the support team's existing review queue.
Built With
- css
- ezio
- html
- javascript
- python
- rest-api
- sha
