Pactpay

Let your AI agent shop within your rules. PayPal holds the money until it's right, while AI inspects checkout evidence and deterministic safeguards verify every constraint before capture or void.

Comment

Inspiration

I kept coming back to one question while exploring the future of AI agents: What happens when an AI agent is allowed to spend someone else's money, but the deal changes just before the payment goes through?

Imagine asking an agent to find a wireless mouse under $50. It finds one, gets your approval and proceeds to checkout. But somewhere along the way, the seller adds an unexpected fee, the price changes, the product is substituted, or the checkout page contains hidden instructions telling the agent to ignore your budget.

The agent may still complete the purchase successfully, but it is no longer the purchase you agreed to.

I realized that most agentic shopping experiences focus on helping agents discover products and complete transactions. I wanted to explore the problem on the other side: How can people delegate purchasing to AI without having to surrender control over their money?

That question became PactPay. My goal was to build a layer that allows AI agents to shop on someone's behalf while ensuring that the final payment remains governed by the user's original intent.

What it does

PactPay is a policy-enforcement layer between AI shopping agents and PayPal. It separates the intelligence that discovers and recommends a purchase from the authority that decides whether money can actually move.

Here's how it works:

  1. Express your intent: I built a natural-language and voice-based interface where users can describe what they want to purchase. Meta Muse converts the request into structured purchasing rules covering budgets, currencies, mandatory requirements and preferences. Users review and approve these rules before shopping begins.

  2. Let the agent shop: A tool-calling AI agent searches live product offers through Channel3, evaluates them against the approved rules and selects eligible products. The agent operates through constrained tools and cannot change approved rules or independently release funds.

  3. Authorize, don't immediately capture: PactPay creates a PayPal Sandbox order using the AUTHORIZE intent. The user approves the payment on PayPal, placing the funds on hold without capturing them.

  4. Inspect before settlement: PactPay generates checkout evidence from the Sandbox transaction's stored merchant terms and subjects it to two-model inspection when both providers are configured. Meta Muse Spark 1.3 and Muse Spark Open independently examine the checkout screenshot. Deterministic code validates the structured readings, checks the arithmetic, compares charges with the approved rules and identifies suspicious instructions directed at AI agents.

  5. Enforce the decision: A server-side settlement gate independently revalidates the live Channel3 offer, PayPal authorization, approved purchasing rules and stored inspection verdict. If every required check passes, it captures the authorization once. If anything violates the rules, is stale or cannot be verified, it voids the authorization instead.

The result is a system with three possible inspection outcomes: Safe to pay, Do not pay, and Can't verify. Uncertainty is never treated as approval.

I also built PactPay Guard, a standalone Chrome Manifest V3 extension that brings checkout inspection to third-party shopping pages. It combines DOM and CSS-based inspection with optional visual AI analysis and intervenes before a user or AI-driven browser proceeds with an unsafe or unverifiable checkout.

How we built it

I built PactPay independently, from the initial architecture through the full-stack application, payment integrations, browser extension, testing and deployment.

The most important architectural decision I made was to separate probabilistic AI reasoning from deterministic financial execution. Models can interpret requests, generate rules, call tools and inspect visual evidence, but they never have authority to change approved rules or make the final capture decision.

AI models and provider integration

During initial development, I integrated Meta Muse models extensively:

  • Muse Spark 1.3 through the Meta Model API: Used for natural-language purchasing rule generation, structured outputs, agentic tool calling, product analysis and checkout vision.
  • Muse Spark Open through Baseten: Integrated as an alternative checkout reader to explore independent visual verification and an open-weights inference path.
  • Muse Voice Transcribe 1.0: Used for voice-based purchasing requests.

The recorded product walkthrough demonstrates this earlier Muse-powered implementation.

During final deployment, I encountered a provider-specific availability issue: requests to Muse Spark 1.3 from the hosted Render environment returned a 404 model-not-found response, even though model discovery using my local Meta API credentials listed the model as available. I could not rely on the same model configuration behaving consistently across local and hosted environments.

Rather than compromise the product or remove its AI capabilities, I adapted the provider integration and migrated the runtime configuration to Groq:

  • GPT-OSS 20B for natural-language processing, structured rule generation and agentic tool workflows.
  • Qwen 3.8 27B for visual checkout inspection.
  • Whisper Large V3 Turbo for voice transcription.

This migration also gave me an opportunity to make the AI provider layer more interchangeable while keeping the core purchase-policy and settlement logic independent of any particular model vendor.

For payments, I integrated PayPal's Sandbox Orders and Payments APIs to implement authorization, capture, void, transaction read-back and reconciliation, alongside signed webhook verification and idempotency keys.

I used Channel3 for live product discovery and offer revalidation, AG Grid for transaction and activity dashboards, and Render for deployment.

The application is built with Next.js, React, TypeScript and PostgreSQL. I developed a shared TypeScript core containing the policy engine, settlement gate, checkout inspection, constrained agent loop, autonomy controls and hash-chained audit functionality. I also built the browser extension separately using Chrome Manifest V3.

I developed automated tests around the core and extension, with 245 core tests and 31 extension tests documented in the project.

Challenges we ran into

The hardest part was not getting an AI model to recommend a product. It was making sure that an incorrect AI decision could not directly result in an unauthorized payment.

Separating AI intelligence from financial authority: LLMs can misunderstand instructions, produce inconsistent outputs or be influenced by malicious page content. I had to design a deterministic settlement gate that independently validates every important condition instead of trusting the model's verdict.

Handling the PayPal payment lifecycle: Authorization, capture, void, asynchronous webhooks and uncertain API outcomes require careful state management. I implemented explicit transaction states, idempotency keys and reconciliation to prevent duplicate captures and unsafe retries.

Detecting checkout manipulation: Important information may be hidden in HTML, rendered using CSS, or visible only in a screenshot. I combined programmatic page inspection with visual AI analysis, while ensuring that missing evidence or conflicting readings result in an inconclusive verdict rather than an unsafe approval.

Building a constrained agent: I wanted the agent to be useful and capable of making decisions about which tools to call, without being able to modify purchasing rules, increase spending limits or release funds. Designing this boundary required careful tool permissions and server-side enforcement.

Making the system testable: Payment failures and security scenarios are difficult to demonstrate consistently. I built clearly labelled hostile-checkout simulations and repeatable test scenarios so that both successful and rejected settlement paths could be evaluated using PayPal Sandbox.

AI provider availability and deployment migration: One unexpected challenge emerged when moving the application from local development to its hosted environment. My Meta Muse integration worked during development, but requests for Muse Spark 1.3 from Render started returning a 404 model-not-found response. Interestingly, the model appeared in the model listing when I checked using my local Meta API credentials, so the issue was not something I could resolve simply by changing frontend code or rebuilding Next.js. I had to investigate the provider configuration, distinguish local API access from hosted access, and adapt the application to use a different inference provider. I migrated the text, vision and transcription workloads to separate Groq-hosted models while preserving the existing policy engine, constrained agent architecture and deterministic settlement gate. This taught me to treat model-provider availability as an infrastructure dependency and to keep critical application logic independent of the inference vendor.

Accomplishments that we're proud of

  • Built a complete PayPal Sandbox authorization-before-capture lifecycle, including actual authorization, capture and void operations.
  • Implemented a deterministic, server-side settlement gate that independently checks the approved Purchase Pact, fresh catalog information, PayPal transaction details and checkout inspection evidence.
  • Integrated multiple Meta Muse capabilities across natural-language understanding, agentic tool calling, voice transcription and multimodal checkout inspection.
  • Built a two-model checkout verification path that requires agreement before a transaction can receive a Safe to Pay verdict.
  • Integrated Channel3 for live product discovery and offer revalidation at settlement.
  • Developed PactPay Guard, a separate browser extension that examines checkout pages for hidden instructions, unexpected charges and visual inconsistencies.
  • Built an MCP server and HTTP gateway so external agents can interact with PactPay through limited, expiring and revocable keys.
  • Implemented a hash-chained audit trail and an AG Grid-based Control Center for examining transaction and policy decisions.
  • Established an automated testing suite covering 245 core tests and 31 extension tests.
  • Integrated multiple AI capabilities across natural-language understanding, structured outputs, agentic tool calling, voice transcription and multimodal checkout inspection, initially using Meta Muse and subsequently migrating the hosted runtime to Groq.
  • Built an independent checkout inspection workflow with separately prompted readings, deterministic validation and explicit handling of conflicting or insufficient evidence.
  • Designed the AI integration so that model-provider changes do not give models additional authority over approved purchasing rules or PayPal settlement decisions.

What I am most proud of is that PactPay goes beyond demonstrating that an AI agent can initiate a payment. It demonstrates a complete decision process in which a payment can be authorized, independently inspected and either captured or safely voided based on the user's approved rules.

What we learned

Building PactPay changed how I think about developing AI-powered applications.

I learned that integrating an LLM into an application is very different from engineering a system that can safely act in the real world. A model can provide useful reasoning, but that does not mean it should be trusted with unrestricted execution authority.

I gained hands-on experience with agentic tool calling, multimodal models, structured outputs, voice interfaces, prompt-injection risks, payment APIs, transaction state machines, idempotency, webhook verification and browser security.

The biggest lesson for me was that trustworthy AI is not achieved by making a model responsible for everything. It comes from designing the right boundaries around what the model is allowed to do.

I also learned a great deal about building and testing a fairly complex product independently, from system architecture and backend engineering to frontend experience, infrastructure and security.

What's next for PactPay

I see PactPay as an early exploration of a broader payment-control infrastructure for agentic commerce.

There are several directions I would like to pursue:

  • Merchant-verified checkout evidence: Introduce merchant-side attestations so that stores can cryptographically sign the product, price and charges they intend to collect.
  • Broader retailer compatibility: Improve PactPay Guard's DOM and visual inspection across different retailer checkout layouts, including more complex carts and dynamic pricing.
  • Agent identity and authorization standards: Explore integration with emerging agent-payment standards such as Google's Agent Payments Protocol (AP2), while keeping PactPay's post-authorization verification layer complementary to them.
  • Multi-currency and multiple catalog providers: Expand beyond the current USD demonstration flows and single catalog integration.
  • Developer adoption: Make the MCP and HTTP gateway easier for external agent platforms, marketplaces and procurement systems to integrate.

Ultimately, I want to explore how PactPay could evolve from a working consumer-facing prototype into an embeddable policy-enforcement layer that helps agent platforms let AI transact while keeping users in control.

My belief is simple: AI agents should be able to earn the freedom to shop, without ever receiving unrestricted authority over someone's money.

Built With

Share this project:

Updates

Submission history