Inspiration
AI shopping agents are starting to make purchases on people's behalf. However, online deals are not always what they appear to be. A low advertised price can hide an automatic renewal, a "free returns" promise can come with a restocking fee, and important details are often buried in the fine print.
I wanted to explore a safer approach: what if every purchase an AI agent makes had to be checked before any money moved? That idea inspired Countersign, a payment safeguard for AI shopping agents built for the PayPal AI Hackathon.
What it does
Countersign sits between a shopping agent and PayPal. The buyer sets a budget and a few rules, such as no subscriptions, free returns only, and no fees beyond the advertised price. PayPal places a hold for the budget, and every offer the agent proposes is reviewed before any payment is captured.
Clear-cut cases are handled by rule-based code. If an offer breaks one of the buyer's rules, it is rejected immediately without an AI call. If an offer passes the rules but still looks questionable, an AI model reads the fine print. For genuinely unclear cases, Countersign runs a short "trial": a prosecutor argues against the purchase, a defense argues for it, and a judge makes the final decision. Every argument must cite the specific line of evidence it relies on.
When an offer is approved, PayPal captures exactly that amount and releases the rest of the hold. If no offer passes, the hold is voided. Each case is saved as a hash-chained receipt, so any later change to an old record can be detected.
The web app includes four views: a Courtroom that shows each case step by step, a Receipts page with chain verification, a Dashboard with spending and savings totals, and an Evaluation page with accuracy results.
How we built it
I developed Countersign using a Python backend and a vanilla HTML, CSS, and JavaScript frontend. The backend relies on the Python standard library for HTTP and concurrency, using urllib for API requests, ThreadingHTTPServer for the web server, and concurrent.futures for parallel work, with Pydantic for schema validation and Pytest for testing.
The payment flow is built on the PayPal Orders v2 API. The backend creates an order with an authorize intent for the buyer's full budget, the buyer approves it through the PayPal JavaScript SDK button, and the server authorizes the order to place a hold. Once a verdict is reached, the server either captures the exact approved amount and releases the remainder, or voids the authorization entirely, so no funds move until a purchase is approved.
Before any AI is involved, every candidate offer receives a deterministic risk score from 0 to 100. Merchants with a verified case file lower the score, while unverified merchants, large purchases relative to the budget, regex signals in the fine print such as recurring-charge language, teaser pricing, fee language, and return costs, and prior rejections of the same merchant all raise it. The score places each offer in a low, medium, or high tier, which determines how much scrutiny it receives. Low-risk offers from verified merchants are checked by code only, and any offer that fails a hard rule on budget, subscriptions, return cost, or hidden fees is rejected without an AI call. Medium-risk offers receive one quick review, and a critical finding escalates them to a full trial, while high-risk offers go directly to trial, where a prosecutor and a defense run concurrently and a judge issues the verdict. Code always has the final word, so if any hard rule failed, an approval from the judge is overridden.
I also implemented this decision flow as a LangGraph StateGraph. Routing by risk tier, the hard-rule check, the quick review, and each part of the trial are separate nodes that share a typed state, and conditional edges choose the next node from the risk tier, the rule results, and the quick review's findings. The prosecutor and defense nodes fan out from the trial node and run in parallel, then fan back in to the judge node. The graph is a drop-in replacement for my original orchestration function, and a comparison script runs both versions side by side to confirm they produce the same verdict, the same deciding stage, and the same sequence of events for all 40 labeled offers and all three demo scenarios.
The AI agents use the Anthropic Messages API, with a fast model handling fact extraction and arguments and a stronger model acting as the judge. The extractor converts the product page, checkout, and terms into structured facts, including the total due today, any recurring charge and its interval, the return cost, and hidden fees, each linked to evidence IDs. The first-year cost is then calculated in code from those facts. A safety net merges the model's output with a regex-based extractor and always keeps the more cautious value, so the model can never miss a trap that pattern matching would have caught.
Alongside my direct API client, I built a LangChain model client using ChatAnthropic from langchain-anthropic. It implements the same interface as the original client, returning the reply text along with input and output token counts from LangChain's usage metadata, so the extractor and every agent can run through LangChain without any changes to their code. A smoke test runs fact extraction and a quick review on a real offer through this client.
Every model reply must be valid JSON that matches a Pydantic schema. If a reply fails validation, the error is sent back to the model for one corrective retry. Each argument's evidence IDs are checked against the offer's real evidence lines, and any point without a valid citation is discarded. If a call errors, times out after 15 seconds, or fails validation twice, that step falls back to an offline analyzer so a case always completes.
To keep the system fast and inexpensive, fine print for all candidate offers is extracted in parallel with a thread pool, so the wait is close to a single model call rather than one per offer. Extracted facts are cached under a SHA-256 fingerprint of the offer text, which means unchanged terms are never read twice and any edit to the terms automatically invalidates the cache. Every call also records input and output tokens, latency, and cost for each agent.
Each completed case is stored as a receipt containing the SHA-256 hash of the previous receipt, creating a tamper-evident chain that pinpoints the exact record if anything is edited later. The server streams each step of a case to the browser as newline-delimited JSON, so the courtroom view updates live as the case runs.
The project includes 46 core Pytest tests, including tests that run the real PayPal and model client code against mock servers to verify URLs, headers, amounts, and token accounting, plus 40 parametrized tests that check the LangGraph version against the original on every labeled offer. I also built an evaluation set of 40 labeled offers, split into 20 for tuning and 20 held out for final testing, which measures traps approved, good offers blocked, extraction accuracy, fallback rate, latency, and consistency across repeated runs. A separate benchmark compares a naive design against the optimized pipeline on AI calls and response time. My technology stack includes Python, Pydantic, Pytest, LangGraph, LangChain, HTML, CSS, JavaScript, the PayPal Orders v2 API, the PayPal JavaScript SDK, and the Anthropic Messages API.
Challenges we ran into
One of my biggest challenges was getting reliable structured output from the AI model. Its responses exceeded my original 500-token limit, which cut off the JSON and caused every call to silently fall back to offline mode. A smoke test exposed the issue, and raising the limit resolved it.
I also ran into SSL certificate errors when connecting to the APIs from Python on macOS, and a PayPal error during my first live benchmark because the automated runs created holds that no buyer had approved. I resolved these by installing the correct certificates and running the benchmark with simulated payments while testing the real PayPal flow separately.
Another challenge was a trap that kept getting approved: a backpack advertised with a "lifetime warranty" whose terms limited it to 90 days. Because it involved no extra cost, the model treated it as minor. I added a general rule that when the fine print withdraws a promise made on the product page, the issue is critical.
Adding LangGraph and LangChain late in the project was a challenge of its own, because I did not want them to change any results I had already measured. I built each one on a separate Git branch, wrote a script that runs the original and LangGraph versions side by side, and only merged once every verdict matched.
Accomplishments that we're proud of
I successfully built a working hold, capture, and void flow in the PayPal sandbox, along with a full web app and 86 automated tests.
On my development set, the offline checker alone approved 4 traps, while the AI agents approved 0 traps and blocked 0 good offers. On the held-out test set of 20 offers run 3 times each, one evaluation approved 0 traps and another approved 1, with no changes in between, and no good offers were blocked. Each offer cost about $0.002 in model usage and took about 4 seconds.
By letting code handle the clear cases, I reduced the blood pressure monitor scenario from 14 AI calls and 46 seconds to 6 calls and 20 seconds, and to 3 calls and 13 seconds with caching, while reaching the same decisions. These are results from my own evaluation set, not a statistically validated benchmark. I am especially proud that every decision is explained with the evidence behind it.
What we learned
This project helped me understand how to combine rule-based logic with AI reasoning in a system that handles real payments. I gained hands-on experience with the PayPal Orders API, payment authorization and capture, structured AI outputs, LangGraph orchestration, and building evaluation sets.
I also learned that most purchasing decisions do not need AI at all, and that letting code handle the clear cases makes a system faster, cheaper, and easier to trust. Rebuilding the pipeline in LangGraph showed me how naturally it maps to a graph, with conditional routing for scrutiny tiers and a parallel fan-out for the trial. Finally, I learned to rely on a held-out test set, since it gives a more honest picture than the data used for tuning.
What's next for Countersign
My next goal is to test Countersign with real merchant data instead of demo stores. I also want to expand the evaluation set with more types of traps, let buyers' past decisions shape their rules, and support more PayPal payment flows. My long-term vision is to make AI agent purchases safer and more transparent for the people who rely on them. Countersign: your agent signs, Countersign checks the deal.
