Nkap Agent

An AI agent that investigates mobile-money payments that don't match the ledger, and can only propose the fix. Fixed rules check it, and a different key must approve.

Comment

Inspiration

With mobile money, a payment, the operator’s statement and the merchant’s ledger sometimes disagree: a fee that was never recorded, a payment posted twice, money that arrives with no owner. Today a person digs through three sources to find out why, then fixes the ledger by hand.

The questions behind this project came from building a live mobile-money product, where reconciling those three sources is somebody’s evening, over and over. That is why I build NkapNkap, an open mobile-money gateway with a real double-entry ledger, and why I asked it a narrow question: can a model do the digging, without ever being trusted with the money?

What it does

Nkap Agent reads a payment, the operator's statement and the ledger with four read-only tools. It names one of Nkap's cause hypotheses (an unrecorded fee, a duplicate posting, …) or declines when the evidence does not support one. If it has a hypothesis, it proposes a ledger entry that corrects it. It cannot approve it.

The model proposes, fixed rules check, a different key approves.

  • The model (NVIDIA Nemotron 3 Super 120B on Nebius Token Factory) answers with one strict JSON object per turn. It never writes a request itself.
  • A deterministic validator, with no model in it, checks the accounts and the currency, that the postings balance and stay under an amount ceiling, and that every value the model cites really exists in the recorded trace and contains the entry's amount.
  • The proposal reaches Nkap with the agent's key. A person approves it with a different key. If the agent's own key tries to approve, Nkap refuses it (HTTP 403, adjustment-self-approval). You can try this on the live demo.

The validator cannot tell a true hypothesis from a plausible one. The human approver is the last check, and we say so on the page and in the README.

Live demo: https://demo.nkap.dev. It has three clearly labelled modes: a recording of a published run (no model), scripted answers, not a model (the real tools, validator and Nkap, with the expected answer instead of a model), and live (the model, under a daily budget). Pick a case, press Run, then approve as the other key, or watch the agent's own key get refused.

How we built it

  • Java 21 and a small JDK HTTP service. The page is plain HTML, CSS and JavaScript under a strict content security policy: it only uses textContent, loads only its own files, and keeps nothing in the browser.
  • NVIDIA nvidia/nemotron-3-super-120b-a12b through Nebius Token Factory's OpenAI-compatible API, with strict json_schema output.
  • The model cites evidence as pointers (tool call id and JSON path). The harness reads the value from the recorded trace, so the model's own rendering of a value is never what gets checked. Tool results are wrapped as untrusted text.
  • The proposing key and the approving key are held by different classes, and the agent's tools can only GET.
  • Docker Compose for the stack, a Cloudflare tunnel in front of a small VPS, a daily and a per-visitor cap on live runs that fails closed (no key, no state file, no live run).
  • Open source, Apache-2.0: https://github.com/Deval123/nkap-agent

Challenges we ran into

  • Reasoning tokens are billed as completion tokens. With max_tokens at 500, 3 of 3 requests ended with no answer. The loop uses 6000.
  • The provider enforces the shape of the schema, not the coupling between fields. Even when asked for prose, 20 of 20 answers were valid JSON, with unused fields filled with empty values. Our parser reads only the fields an action uses.
  • Our first run matched 10 of 18. The records showed three causes that were not the model's fault: the fee was not in what the tools returned, the pattern of a correction was read as a copy of the posting, and a correct refusal was rejected because it cited a null value as its evidence.
  • Showing the proof. The agent must be convincing without being believed, so every step of the page names its mode in words.

Accomplishments that we're proud of

  • The separation of powers is real and testable: the agent's own key is refused by Nkap, live, in front of the visitor.
  • After reading the first run's failures and giving the agent a tool that shows statement lines with their fees, 17 of 18 runs matched the expected outcome (118,122 prompt and 27,392 completion tokens, about 0.06 USD for the 18 runs). The cases that need no proposal (money without an owner, a prompt injection, "I do not know") matched 3 of 3 each.
  • We say what the numbers are not: the prompt was tuned after the first run, so the six cases are a demonstration set, not a benchmark.
  • The README names every weakness of the design, with what keeps it harmless and what would end it.

What we learned

  • With a reasoning model, budget the output tokens for the reasoning, not for the answer.
  • Strict schemas make the shape reliable, not the meaning. Validation belongs in the harness, and the model's output should only ever be a proposal.
  • Giving the model the missing data worked better than rewriting the instructions: the biggest jump (10 to 17 of 18) came from one new read-only tool.

What's next for Nkap Agent

  • Nkap 2.2.0 will ship the adjustment API. It is already merged on main and is not specific to this project: an escalated payment used to be a dead end, and a human can now finish what the reconciler could not, by proposal and approval. The agent is one caller of it, not part of it.
  • A case set built independently of the prompt. The rates reported here come from the scenarios this demo ships, and the prompt was tuned after the first run. The next measurement that means anything is a larger set of reconciliation cases written without looking at the prompt, run repeatedly, and published with the model and date beside each number, including the cases where the model is wrong and the server stops it.
  • Separation of duties by key scope. ADR 0017 names the weakness this design accepts: the API key carries one admin boolean, which also gates the reads a proposer legitimately needs, so one boolean cannot tell a proposer from an approver. What separates them today is key identity and the server's refusal to let a key decide its own proposal. Scopes on the key are the stronger answer and the next step.

Built With

Share this project:

Updates

Submission history