ResolveOS

ResolveOS is an AI copilot for payment operations.

Comment

Inspiration

Payment failures are rarely clean. A capture times out, but the money may have moved anyway. A retry that looks harmless can charge a customer twice, and a settlement file can be half an hour out of date and still disagree with the ledger.

Operations teams make these calls under pressure, jumping between processor dashboards, ledgers and runbooks. AI could speed this up, but an AI that can move money on its own is a frightening idea. That tension inspired ResolveOS: how do you get an AI's speed without trusting it with the final decision?

What it does

ResolveOS is an AI copilot for payment operations. When a payment fails, stalls or gets held, it:

  1. Deduplicates the event and classifies the exception
  2. Gathers fresh evidence from the processor and ledger
  3. Retrieves relevant policies, runbooks and past cases
  4. Produces a diagnosis and three ranked actions, each with a risk score
  5. Runs every action through a deterministic safety gate

The gate routes each case to one of four outcomes: auto-run, human approval, more evidence or hard block. Compliance holds can never be released by the agent, and money never moves without a person's approval.

If an operator rejects a plan, ResolveOS proposes a safer one and shows what changed. After three rejections, the case goes to its owner. If a late event changes the payment's state, stale actions are cancelled. For example, a pending retry is invalidated when the processor confirms the original capture succeeded.

Approved actions run exactly once, the real state is re-read to verify the result, and every step is recorded in an audit trail you can replay. All payments in the demo are simulated.

How we built it

Frontend: React, Vite and Tailwind, with Recharts for metrics and WebSockets for live updates. The interface is inspired by Apple's design resources: frosted bars, large type and a vertical sidebar.

Backend: FastAPI, SQLAlchemy and Alembic, with LangGraph coordinating the investigation pipeline. It runs on SQLite out of the box, with PostgreSQL + pgvector and Redis supported through configuration.

AI and ML:

  • Nemotron via the NVIDIA API for structured diagnosis, with output validated against a strict schema
  • Hybrid RAG over 2,414 knowledge chunks, combining BGE embeddings and TF-IDF with reciprocal-rank fusion:

$$ \text{RRF}(d) = \sum_{r \in {\text{dense},\ \text{lexical}}} \frac{1}{k + \text{rank}_r(d)}, \qquad k = 60 $$

  • A logistic-regression triage model on BGE embeddings
  • A LightGBM action predictor
  • A calibrated LightGBM risk model that scores each proposed action $a$ separately, given case features $x$:

$$ p_a = g\big(f(x, a)\big) $$

Here $f$ is the LightGBM model and $g$ is an isotonic calibrator.

The core design rule is that models recommend and code decides. The gate is plain Python with no model calls. It runs when a plan is created, again at approval and again immediately before execution. An action can only run automatically when all of these hold:

$$ \text{allowlisted}(a) \;\land\; \neg\,\text{movesMoney}(a) \;\land\; \text{evidenceFresh} \;\land\; \big(\text{score}(a) \le \tau \;\lor\; \text{riskExempt}(a)\big) $$

The risk-exempt actions are the ones with no side effects, such as escalating to compliance.

Challenges we ran into

  • Connecting a trained model to a live system. The notebook used different names for exception types, processor states, rails and actions, so we built a translation layer. One input, amount_rel, wasn't stored anywhere, so we inferred it from the model's split points.
  • Mapping meaning, not just labels. We first sent "reconcile" to the risk model as RECONCILE_FIRST, which scored it as high risk after a late success. In that situation it really means CANCEL_PENDING_AND_VERIFY.
  • Respecting the model when it was inconvenient. The real risk model rated some actions as medium risk that our earlier heuristic had auto-run, which added an approval click to our demo. We kept that behaviour rather than adjusting inputs to get a nicer result.
  • Models disagreeing with the plan. On the timeout case, the action predictor favours "reconcile first", while our scripted fallback diagnoser recommends a retry. We chose to show that disagreement in the UI.
  • Prompt injection. Retrieved documents and notes are untrusted input, so we planted a malicious merchant note in the knowledge base to prove it gets quarantined.
  • Test isolation. A background thread that loaded the models leaked into unrelated tests, producing confusing failures until we traced it.

Accomplishments that we're proud of

  • The full demo works end to end: a timeout arrives, the retry is rejected, "reconcile first" is proposed, a late success cancels the stale retry, and the final state is verified and replayable
  • 0 unsafe autonomous actions across our 10 synthetic scenarios, compared with 7 for a rules-only approach and 5 for an agent without the gate
  • 29 passing backend tests covering duplicate events, stale evidence, role enforcement, repeated execution and the kill switch
  • Every degraded mode is labelled, so the app never presents a fallback as a real model result

The baseline comparison uses synthetic scenarios and an oracle we defined, so it shows the gate's value by design. It is not a measurement on real payment traffic.

What we learned

  • Safety belongs outside the model. Prompting can reduce bad output, but only deterministic code can guarantee that a hold is never released.
  • Near-perfect test scores can mislead. Our triage model reached a macro-F1 of 1.0 on the synthetic test set, but 0.76 on the harder set and 61% accuracy on a small real-text held-out set ($n = 18$, with model-generated labels).
  • Verification matters as much as execution. Re-reading state after every action caught a bug where a valid next-day resubmission was being marked as failed.
  • Being honest about data builds trust. Labelling results as measured, synthetic or offline made our metrics easier to defend.

What's next for ResolveOS

  • Evaluate Nemotron-backed diagnosis live in the app, with an NVIDIA API key connected
  • Let the action predictor help rank options, after more evaluation
  • Retrain and audit the models on human-reviewed labels and real operations data
  • Connect to a processor sandbox in place of the simulator
  • Add web push so approvers can respond from their phones
  • Add dollar cost tracking per case, using real token prices

Built With

Share this project:

Updates

Submission history