• 

• 

• 

  Score

About the project

False Promises is our Approval Gate for the Ezio AI Decision-Grade AI Buildathon. We wrap a local simulated Ezio API with an independent policy checker so a Lumen Cloud support lead can APPROVE, HOLD_FOR_REVIEW, or BLOCK a customer email before the mock send runs.

Pitch: before AI sends a promise, we look for a customer who makes it false.

Persona / action: customer support / send_email
Run the demo: python3 gate.py serve --persona customer_support
Wrapper: Python 3.9+, standard library; python3 wrapper/serve.py

Inspiration

Trust scores are easy to over-trust. In this challenge, a GREEN overall tier with accuracy not_scored can still sit on top of a private company promise the mock never sees: prices, loyalty tenure, refund windows, trials. The starter wrapper mostly followed the aggregate score, so most bad drafts would have gone out.

We were inspired by a support reality that is older than any model: a correct number can still be an impossible promise. “You qualify for our 15% loyalty discount on the Pro plan” is true for a Pro customer at 12 months and false at 11. The amount matches policy. The customer does not. If the email still sends, the company has made a false promise.

That became our build rule. Do not invent a second scoring engine. Do not hard-code practice sentences. Call Ezio over HTTP for the signals it actually has, then find the exception: a concrete customer, drawn from policy, who makes the claim false. If we cannot find one in the modeled scope, we say so honestly — that is not a proof that every word is true.

What it does

The wrapper is a local decision gate for fictional Lumen Cloud customer email.

1. A support lead pastes a draft and (when it matters) the customer’s plan and tenure.
2. The wrapper calls the simulated Ezio API at EZIO_URL for verdict, dimensions, and public-knowledge claims.
3. Independently, it checks price_sheet.csv and policies.json: prices, loyalty and annual discounts, stacking, refunds, trials, first-response times, and unauthorized product or exception promises.
4. For discount language it searches for a counterexample. Personalized offers test the actual customer. Broader offers (“every customer,” “all subscribers”) search policy-boundary customers — including the loyalty minimum and the month next to it. A failing witness is recorded with the claim, failed condition, and source file.
5. Policy combines both worlds with BLOCK > HOLD_FOR_REVIEW > APPROVE. Only APPROVE executes the mock email. Held and blocked drafts never send.
6. Every ordinary decision is audited (input, Ezio response, checks, witnesses, reason, evidence, timestamp). Managers can log reject or request revision on a hold; review never overrides the gate or sends mail. A revised draft must go through /decide again.

The persona UI is an offline support desk: draft composer, practice samples, 11- vs 12-month loyalty comparison, decision/evidence panel, session history, and a persistent audit drawer. The browser never scores, never calls the mock, and never runs the action.

Signature demo: the same 15% Pro loyalty offer blocks at 11 months and approves at 12. An overbroad “every customer gets 15% loyalty” returns a concrete ineligible customer instead of a vague confidence drop.

How we built it

Everything stays on the laptop. gate.py starts a private mock Ezio and our wrapper on loopback, sets EZIO_URL, and tears both down. No live Ezio key, no extra packages, no network.

Independent signal. wrapper/source_checks.py normalizes wording (word numbers, percents, punctuation), associates figures with the nearest named plan, and compares extracted facts to the supplied CSV/JSON. It does not look up practice IDs. Discount proofs distinguish personalized vs universal vs conditional scope, prefer a tenure-boundary witness over a cheap “wrong plan” witness, and treat “no counterexample found” as only a modeled-check result.

Decision policy. wrapper/wrapper.py plus wrapper/policy.json: contradicted Ezio claims BLOCK; unverifiable claims HOLD; GREEN is necessary but not sufficient; tone, bias, and agreement each hold below 80; not_scored does not auto-penalize (private claims still get company checks); unavailable and malformed/exhausted Ezio calls HOLD unless an independent contradiction already requires BLOCK. Local checks still run when Ezio is down so we never fail open.

Action gate and records. wrapper/serve.py exposes POST /decide, GET /health, samples, audit, and reviews. mock_actions.send_email runs if and only if the decision is APPROVE. JSONL audit and action logs go to GATE_LOG_DIR.

UI. Plain HTML/CSS/JS under wrapper/ui/, served from the wrapper. Labels and icons carry the risk, not color alone. Loading disables controls; empty and oversized drafts never fire a request.

Verification. Practice on customer support: 16/16 exact decisions, 8/8 bad caught, 0/8 false blocks, 15/15 failure-drill points, action log matching approvals. Contract, harness, source, decision, and review tests pass. Browser checks covered the 11/12 boundary, sample context, review persistence, and mobile layout.

Challenges we ran into

A clean score is not clearance. Many private promises come back GREEN with accuracy not_scored. If we had treated the tier as the decision, we would have shipped false discounts. If we had blocked every not_scored draft, we would have false-blocked routine mail. The hard part was splitting “Ezio cannot see this” from “this is dangerous.”

Find the right exception, not any exception. Searching “every customer” must include ineligible plans and tenure just below the loyalty floor. Searching “you qualify” must use this customer, not a hypothetical Starter user. Claiming a rate without promising it to everyone is not a universal entitlement. Wrong scope looks clever in a demo and fails judging.

One sentence, several facts. A correct Pro price must not hide a wrong first-response time. Trial days, refund windows, and response SLAs all look like “N days.” We had to bind each number to a metric instead of the first duration in the sentence.

Wording that is not the practice set. Word-form numbers, “per cent,” annual totals vs per-month annual prices, negated stacking, and unspecified discount types all had to be general rules. Matching labeled examples would have been a disqualification risk and a brittle demo.

Degraded Ezio. Rate limits, 503s, slow responses, malformed bodies, and unavailable accuracy all had to return a valid decision and a non-empty reason inside the harness timeout, without approving the bad probe after a successful retry. Bounded retries in ezio_client.py plus “known contradiction still BLOCKS” was the combination that held.

UI without leaking the gate. Session history is not the audit log. Comparison must not overwrite the composer’s selected customer. A hold’s “request revision” must persist and must not send. Interrupted fetches had to show an unknown outcome, not a fake BLOCK.

Time. This is a short build window. We froze new features late and spent the remainder on practice, evidence, and the two-minute walkthrough rather than a second persona or a second model.

Accomplishments that we're proud of

• A complete, offline customer-support gate: independent policy checks, Ezio HTTP signals, action executed only on APPROVE, and a persona page judges can type into.
• Find the Exception as a readable proof, not another score: the 11/12-month loyalty pair, plus a witness customer for overbroad offers.
• Practice verification at the target bar: 16/16 exact, 8/8 catch, 0 false blocks, 15/15 failure handling, with actions matching approvals.
• Evidence on every path — approvals included — so “no modeled violation” is explicit instead of empty.
• A hold-for-review workflow that is logged and human, and that never bypasses the gate.
• Policy justifications written from implemented behavior, with thresholds we can defend in two minutes.

What we learned

A scoring API and a source of truth answer different questions. Ezio is useful for public claims and for tone, bias, and agreement even when the overall tier looks fine. It cannot certify Lumen Cloud’s private price sheet. The wrapper’s job is to combine those signals, not replace either one.

We also learned that not finding a counterexample is weaker than finding one. Our records say that modeled discount checks found no witness; they do not claim the entire email is verified. That honesty belongs in the reason string the judge reads.

On the engineering side: keep the contract (HTTP to the mock, never import its internals); keep checks inside the 60-second decision budget; treat missing evidence as hold, not approve; and put every decision the UI shows on POST /decide. Tests that only prove the starter HTTP plumbing is up do not prove the business rules. We added source, decision, and review tests for the behavior we actually shipped.

What's next for false-promises-hack-26

• Run the official held-out set when the organizer passphrase is released (python3 gate.py judge --persona customer_support --team "False Promises") and submit the signed results unchanged.
• Extend the same counterexample idea to the paralegal and financial-analyst personas (citations and fact-sheet metrics) without copying customer-support regexes.
• Broaden natural-language coverage — more quantifiers, more billing phrasings — while staying data-driven and free of hard-coded answers.
• Optional stretch after the required path: calibration of dimension thresholds against more independent variants, or a second local check that still cannot replace Ezio or the policy files.
• Keep the product story: challenge the promise, find the exception, then send.

Built With

• codex
• cursor
• ezio