Inspiration

Teams in healthcare, finance and legal want to use AI on their documents, but they cannot paste patient records or loan files into a system they cannot inspect. Most "private AI" tools just ask you to trust them. We wanted one that lets you verify.

What it does

VaultAI redacts personal data (names, SSNs, emails, phone numbers, addresses, IBANs, card numbers, dates and labelled IDs) on our server before any model sees the text, and shows a side-by-side preview for approval. Questions are answered by NVIDIA Nemotron Super with numbered citations. A second model, Nemotron Ultra, acts as an independent judge: it splits each answer into claims and marks each as supported or unsupported by the sources. If the first check scores low, the answer is regenerated once with a stricter prompt and re-checked. User questions are redacted too.

You can also inspect everything: a Wire Inspector shows the exact payload sent to the model, a hash-chained audit log records every action and can be integrity-checked, and a Purge button clears the session and returns a receipt.

How we built it

Next.js and TypeScript, deployed on Vercel. Everything model-related runs through Nebius Token Factory using the standard openai package with a changed base URL. We route each task to the smallest suitable model:

Nemotron Nano for query cleanup Nemotron Super for grounded, cited answers Nemotron Ultra as the judge Qwen3-Embedding-8B for embeddings (the only embedding model listed on Token Factory)

Session data (chunks, vectors, audit log) lives in the browser, so the server is stateless and Purge means something real. Retrieval combines vector search with BM25 keyword search and falls back to keywords if embedding is slow. A shared prompt file keeps the app and the evaluation measuring the same thing. There is also an offline mock mode, 18 unit tests and 12 API smoke checks.

Challenges we ran into

Reasoning models spend max_tokens on thinking, which produced empty answers and truncated judge output until we raised limits and added a retry. Embedding latency varied from under a second to over ten, so we added keyword search and a timeout fallback. Judge output sometimes arrived as malformed JSON, and the models sometimes cited with fullwidth brackets. We validate, retry and normalise both. Our first evaluation prompts had drifted from the app's prompts, so the numbers did not describe the app. We moved every prompt into one shared file.

Accomplishments that we're proud of

In our final evaluation (Oct 29), all 13 questions passed, including a document with an embedded prompt-injection instruction (ignored), a question that needs two documents, and an amendment that overrides an earlier value. The Ultra judge flagged all 7 deliberately wrong answers, including a dose with the wrong frequency and a previous lab value presented as current. A full evaluation run costs about $0.058. We report this as a small synthetic smoke test, not a benchmark.

What we learned

Verification is more useful than a confidence score: showing which claim is unsupported changes how much you trust the rest. We also learned that model "reasoning" output needs generous token budgets, and that you should measure the app you ship, not a parallel copy of it.

What's next for VaultAI

A stronger redaction model for names in plain prose, OCR for scanned PDFs, an NVIDIA embedding model when one is available on Token Factory, and larger, real-world evaluations

Honest limits

Redaction is rule-based and misses unlabelled names; there is no OCR; the judge is a model and can be wrong; the demo data is synthetic. These are also listed in the README.

Built With

  • bm25
  • nebius-token-factory
  • nemotron-3-nano
  • nemotron-3-super
  • nemotron-3-ultra
  • next.js
  • node.js
  • nvidia-nemotron
  • openai-sdk
  • pdf-parsing
  • qwen3-embedding
  • rag
  • react
  • tailwindcss
  • typescript
  • vercel
  • web-cypto
  • zod
Share this project:

Updates

Submission history