MarginGuard

Check the fix before you ship it: MarginGuard tests AI margin advice against live official policy via Tavily, NVIDIA Nemotron on Nebius plans and judges, and code verifies every quote.

Comment

Inspiration

Small online sellers get margin alerts and generic advice ("tighten your discount combination settings"). On Shopify that advice is sometimes wrong: custom discounts that staff apply manually in POS or on a draft order apply regardless of combination settings, so the toggle changes nothing and the leak continues. An assistant that sounds confident but ignores the platform's own rules costs money. We wanted an agent that treats official policy as a test the proposed fix has to pass.

What it does

  1. Finds the leak with exact math. Import an order CSV. Deterministic code computes contribution in integer cents and flags discount stacking, shipping overruns and return spikes. Every finding shows its formula and source rows; seller assumptions (discount ceiling, shipping allowance) are editable.
  2. Challenges the proposed fix with live policy. For the flagged issue, MarginGuard sends Tavily Search a neutral query for the seller's platform and context, domain-restricted to help.shopify.com, woocommerce.com, usps.com or ups.com, then Tavily Extract pulls the official page so the rule engine reads full sentences. On the Shopify discount-combination pages we reviewed, the result also says if the page has changed since our review. In the demo case, the passage "custom discounts that you apply manually in Shopify POS always apply to the order, even if other discounts have combination restrictions" rebuts the settings fix and redirects the next check to auditing staff/manual overrides — same 28 rows, same $658.08 modeled exposure, different action.
  3. Holds when context is missing. An unknown discount origin, a non-US region, an app whose discount mechanism is unknown, or Shopify Scripts (retired by Shopify on 2026-06-30) puts the recommendation on HOLD whatever the sources say. A retracted or contrary passage also holds; no passage, or conflicting passages, return "not enough evidence".
  4. Drafts a cited check with NVIDIA Nemotron on Nebius Token Factory. Code supplies the policy statement behind each citation; Nemotron-3.5-Lightning writes the hypothesis and a concrete store-operator procedure. Code checks every citation, rejects numbers and causal claims in the prose, and rejects a procedure that only restates the code's check; a held case never reaches the model.
  5. Checks any fix you paste in. Type advice from any AI assistant. Code picks the official site from the fix's wording; Nemotron-3.5-Lightning turns it into a testable claim and search query; Tavily Search + Extract fetch the official page; Nemotron 3 Super judges supported / contradicted / not enough evidence with quotes; code accepts only quotes found verbatim in the Tavily text. On our frozen 8-item list the third run matched 7/8 expected verdicts (the list also guided the fixes, so this is not an independent accuracy claim).
  6. Tests a promotion before launch. In Promo Preflight the seller describes a campaign in plain English; Nemotron 3 Super turns it into rules, code checks them, the seller confirms, and an integer-cent simulator evaluates every basket, shows the worst-case cart and compares it under the original and a reviewed alternative rule.
  7. Watches the policy pages behind each decision. One Tavily Extract call re-reads the reviewed Shopify pages and reports unchanged / changed / unavailable against the fingerprints our decisions were reviewed on.
  8. Exports an audit. Findings, assumptions, sources, decision status and what would disprove the hypothesis export as a report.

How we built it

  • Tavily Search retrieves official policy with neutral queries (never order data), include_domains restricted to the platform's official domain, advanced depth with relevant chunks. Results are screened for domain, relevance, prompt-injection text, retractions, regional scope and duplicates.
  • Tavily Extract then pulls the top official pages, so decisions read complete sentences; Policy watch uses one batched Extract call to re-fingerprint the reviewed pages.
  • Nebius Token Factory serves two NVIDIA models through its OpenAI-compatible API, each verified with GET /v1/models before use and constrained with json_schema structured output:
    • Nemotron 3 Super (nvidia/nemotron-3-super-120b-a12b) judges evidence in Check a fix and turns free-form campaign text into promotion rules (grounded against the seller's own numbers and a local grammar parser).
    • Nemotron-3.5-Lightning (nvidia/Nemotron-3_5-Lightning) plans searches and drafts cited checks in about a second.
  • Deterministic core (TypeScript, shared by an Express server and a Cloudflare Workers build): integer-cent arithmetic, policy applicability rules, verbatim quote verification, and a durable D1 budget ledger (per-minute, daily and lifetime caps per provider) reserved before every provider call. The model never computes money or decides eligibility.
  • React + Vite front end with keyboard support, 320px layouts and reduced-motion handling. 368 unit/API tests and 100 Playwright scenarios.

Challenges we ran into

Real retrieval is messier than fixtures: snippets truncate mid-sentence, two URLs return the same passage, and Nemotron drafts failed our validator in 3 of 5 early live runs (bare citation IDs, a step that repeated the rebutted fix). Switching to json_schema structured output plus code checks made later runs pass on the first call. Our own preregistered benchmark was a negative result: on 24 holdout cases the Tavily package did not improve Nemotron's decisions, mostly because a fixed query found the right page in only 4 cases; after redesigning retrieval (neutral queries + Extract), coverage rose to 8/24, and we publish both results as they came out.

Accomplishments we're proud of

Retrieval changes the decision, not just the citations list: the live Shopify POS passage rebuts the settings fix and turns the next check into an override audit, while an unknown origin holds instead of guessing. Judges can trigger the real Tavily + Nemotron calls themselves on the live demo.

What we learned

The valuable part of an AI margin assistant is knowing when the recommended fix doesn't apply. Keeping arithmetic, eligibility and quote verification in code made the model's job smaller and its output checkable.

What's next

Read discount origin directly from a connected store instead of asking the seller, cover more platform policies, and re-run the preregistered benchmark with human blind review.

Try it

Data and limits

All demo orders are synthetic. Modeled exposure is a review candidate, not realized savings. MarginGuard does not connect to a store or change settings. Check a fix is a model judgment with code-verified quotes, not legal advice. MarginGuard was created during the Submission Period (repository started 2026-10-27).

Built With

Share this project:

Updates

Submission history