VeriDispute AI

Inspiration

Credit card disputes can involve large amounts of evidence, making manual review slow and inconsistent. I built VeriDispute AI to explore how AI could make the dispute resolution process faster, more structured, and explainable.

The core idea was to combine LLM-based evidence extraction, rule-based validation, machine learning, and explainability into one end-to-end system. Instead of simply predicting whether a dispute should be refunded or denied, VeriDispute AI also explains why it reached that decision and routes uncertain cases to human review.

How We Built It

VeriDispute AI is built as a three-part monorepo:

Frontend: React 19, Vite, Tailwind CSS, Recharts, and Axios
Backend: FastAPI, SQLAlchemy, Pydantic, and Uvicorn
ML Pipeline: XGBoost, scikit-learn, SHAP, pandas, and NumPy
LLM: Mistral API for extracting structured information from free-text evidence
Database: SQLite by default, with support for PostgreSQL through DATABASE_URL

The complete resolution workflow is:

A card member files a dispute against a merchant transaction.
The card member and merchant can submit supporting evidence.
The Mistral LLM extracts structured fields such as tracking numbers, delivery status, dates, signatures, amounts, and policy compliance.
A rule engine checks whether the required evidence for the dispute reason has been submitted.
Evidence signals and dispute history are converted into a feature vector.
An XGBoost classifier predicts one of three outcomes: refund, deny, or partial.
SHAP identifies the top factors influencing the prediction.
A counterfactual analysis determines what single feature change could potentially change the outcome.
Confidence-based routing determines whether the case is automatically resolved or sent for human review.
The dispute, evidence, decision, confidence score, and reasoning are persisted and displayed in the dashboard.

Evidence Completeness

The system calculates evidence completeness using:

$$ \text{Evidence Completeness} = \frac{\text{Required Evidence Submitted}} {\text{Required Evidence Types}} \times 100 $$

For example, if a dispute requires two types of evidence and both are submitted:

$$ \frac{2}{2} \times 100 = 100\% $$

This allows the system to distinguish between disputes with strong supporting evidence and those where important information is missing.

What I Learned

The biggest lesson from this project was that building an AI system is not just about training a machine learning model.

I learned how different components can work together to create a complete decision-support pipeline:

LLMs for extracting information from unstructured evidence
Rule engines for deterministic evidence validation
Machine learning for outcome prediction
SHAP for model explainability
Counterfactual analysis for understanding how decisions could change
Confidence-based routing for combining automation with human oversight

I also learned the importance of keeping the training and inference pipelines consistent. The backend must reconstruct the same feature vector and class ordering that were used when training the model.

Challenges We Faced

1. Handling LLM Failures

One challenge was making the system robust when the LLM is unavailable or returns invalid output.

Instead of allowing the entire pipeline to fail, VeriDispute AI gracefully falls back to keyword-based evidence detection. This means the system can still complete the dispute workflow even without a valid Mistral API key.

2. Limited Real-World Training Data

Another challenge was the lack of real historical dispute-resolution data.

To demonstrate the complete ML pipeline, we generated 1,500 synthetic disputes with partially overlapping outcomes and 5% random label noise.

The XGBoost model achieved 71.3% test accuracy using an 80/20 stratified split. Its accuracy can be increased significantly if I have access to previous data of such disputes. The model uses features including:

Tracking presence
Signature confirmation
Merchant response time
Merchant policy compliance
Card member dispute history
Evidence completeness
Dispute reason code

3. Making AI Decisions Explainable

A prediction such as:

refund — 95% confidence

doesn't provide enough information for a human decision-maker.

Therefore, every prediction includes:

Confidence score
Top 3 SHAP factors
Evidence completeness
Human-readable reasoning
Counterfactual explanation

For example, the system can explain that flipping has_tracking from False to True would change the predicted outcome from refund to partial.

4. Handling Uncertain Predictions

We didn't want the system to blindly trust every ML prediction.

Instead, VeriDispute AI uses confidence-based routing:

$$ \text{Route}(c) = \begin{cases} \text{Auto-Reject} & c < 20\% \ \text{Human Review} & 20\% \leq c \leq 85\% \ \text{Auto-Approve} & c > 85\% \end{cases} $$

This creates a balance between automation and human oversight.

Why VeriDispute AI Is Different

VeriDispute AI doesn't treat AI as a black-box decision maker.

It combines:

Unstructured Evidence → LLM Extraction → Rule Validation → ML Prediction → Explainability → Confidence Routing → Resolution

The result is an end-to-end system designed to make dispute resolution more automated, transparent, and explainable.

Limitations

The current version is a prototype and has several limitations:

There is no real authentication system.
The ML model is trained on synthetic rather than real historical dispute data.
Merchant response time currently uses a fixed default of 48 hours.
SQLite is intended primarily for local development and demos.
Keyword-based fallback detection can still be affected by unusual phrasing.

These limitations also define the next steps toward a production-ready system: real dispute datasets, stronger authentication and authorization, production-grade databases, real response-time tracking, and more robust evidence processing.

Overall

VeriDispute AI helped us understand how LLMs, machine learning, deterministic rules, explainability, and human oversight can be combined into a single practical AI workflow.

Rather than building just another prediction model, we built an end-to-end system that turns unstructured dispute evidence into a transparent, confidence-aware resolution decision.