Inspiration
Inspiration PayPal disputes are time-sensitive, evidence-heavy, and expensive for merchants to review manually. Smaller teams may need to inspect an order, tracking details, buyer messages, policy rules, deadlines, and the disputed amount before deciding whether a case is worth fighting.
We built PayPal Dispute Copilot to make that process faster while keeping the merchant in control. The system analyzes the case and drafts a response, but it never submits anything to PayPal until a human explicitly approves it. What it does When a dispute arrives, the Copilot:
- Classifies the dispute reason and deadline urgency.
- Collects available order, tracking, and buyer-message evidence.
- Retrieves the three most relevant policy passages from a local vector store.
- Uses a trained win-probability model to estimate the merchant’s chance of success.
- Calculates expected recovery: [ \text{Expected Recovery} = P(\text{Win}) \times \text{Disputed Amount} ]
- Recommends FIGHT or ACCEPT and drafts a response.
- Checks for prompt injection, PII exposure, oversized inputs, and invalid output.
- Pauses at AWAITING_APPROVAL.
- Submits evidence or accepts the claim only after a reviewer clicks Approve & submit. How we built it The application is a single FastAPI service with a buildless HTML, CSS, and JavaScript interface.
The agent workflow uses LangGraph with five main nodes: Triage → Evidence → Decide → Guard → Submit ↑ Human approval The decision node also contains a tool-selection subgraph. In local AI mode, Qwen3 4B Instruct through Ollama selects from an allowlisted set of tools:
- search_dispute_policies
- get_win_probability Python validates and executes every tool call. The LLM can explain the result and draft the response, but it cannot change the model probability, financial calculation, recommendation, approval requirement, or PayPal action.
For judges who do not have Ollama, the project includes a deterministic offline mode that runs the complete workflow without an API key.
The remaining architecture includes:
- SQLite and SQLAlchemy for disputes, decisions, tool calls, and paused workflow state.
- Redis for atomic webhook event deduplication.
- Chroma for persistent retrieval over 15 local dispute-policy files.
- A scikit-learn logistic regression trained on 200 labeled synthetic disputes.
- PayPal sandbox integrations for OAuth, order creation/capture, dispute retrieval, evidence submission, claim acceptance, and webhook verification.
- Structured JSON logs, request IDs, Prometheus metrics, OpenTelemetry traces, and an optional Grafana/Loki/Jaeger stack. Safety and human control Buyer text is always treated as untrusted input. Security controls are enforced in Python rather than relying only on an LLM prompt.
The guard layer:
- Detects common prompt-injection phrases.
- Redacts PII from logging context.
- Validates structured agent output.
- Rejects oversized payloads.
- Restricts tool selection to an explicit allowlist.
- Requires human review before every submission.
- Adds additional review warnings for high-value disputes. The LangGraph workflow pauses after analysis and persists its state. This means it does not block a server worker while waiting for a reviewer. Challenges we faced Reproducing PayPal disputes in the sandbox Creating a realistic dispute required separate Personal buyer and Business seller sandbox accounts. We completed a sandbox purchase, opened an item-not-received case from the buyer account, and retrieved it using the seller’s REST application.
In our verified test, PayPal’s list-disputes endpoint returned an empty page even though the buyer could see the case. Direct retrieval using the buyer-visible case ID succeeded, so we added a direct case-import path and documented the behavior rather than hiding it. Keeping the LLM away from financial authority Tool calling is useful, but an LLM should not invent a win probability or autonomously decide a financial action. We separated responsibilities:
- The LLM chooses from safe tools, explains results, and drafts text.
- The trained model owns the win probability.
- Python owns calculations, validation, limits, and the final decision rule.
- A human owns submission approval. Making the demo reproducible Local model availability and hardware differ between judges. We therefore implemented two supported paths:
- Ollama with qwen3:4b-instruct for real local model tool calling.
- Deterministic offline mode for a no-key, no-model-download demonstration. Testing security controls honestly We created executable red-team scenarios for prompt injection, forged tracking text, refund escalation, and oversized payloads. The reports are generated from actual runs with guards enabled and disabled rather than using hardcoded results. What we learned We learned that the most reliable agent systems do not give one model unrestricted control. Strong systems combine specialized components:
- LangGraph for orchestration.
- A predictive model for probability.
- Retrieval for policy grounding.
- Deterministic code for financial calculations.
- Guardrails for untrusted content.
- Durable persistence for approval pauses.
- Human judgment for consequential external actions. We also learned that sandbox integrations can behave differently from ideal API documentation. Recording those differences, providing a fallback import path, and clearly separating verified behavior from simulated data made the project more honest and reproducible. What is real and what is simulated The PayPal OAuth and direct dispute-retrieval path was verified using a real PayPal sandbox buyer case. The application genuinely runs the LangGraph workflow, policy retrieval, trained probability model, guards, persistent approval pause, and reviewer interface.
The win-probability model is trained on synthetic labeled disputes because no real merchant dispute-history dataset was available. Merchant order, tracking, and message evidence uses a synthetic evidence store unless connected to a merchant’s systems. Fixture-mode PayPal submissions are clearly labeled as simulated.
No live PayPal credentials, live payments, or production disputes are used.
Built With
- alertmanager
- compose
- css
- docker
- grafana
- jaeger
- javascript
- loki
- opentelemetry
- prometheus
- pytest
