PaperTrail — document in, next step out

Turn a photo of a bill or notice into a deadline, an ordered plan and a calendar file. NVIDIA Nemotron on Nebius Token Factory, grounded by Tavily.

Comment

Inspiration

Bhanu builds Keepary, an OCR document scanner on Google Play. Scanning a document is easy; knowing what to do with it is not. Families miss bill deadlines and government notices because the language is dense and the one date that matters is buried on page two. PaperTrail is the AI brain for that captured document: it reads the paperwork, finds the official page, and turns it into a deadline-ordered plan.

What it does

Upload a photo of a bill, tax notice, rent agreement, bank letter, government form or medical bill. PaperTrail reads it with NVIDIA Nemotron 3.5 Super VL, triages urgency and category with Nemotron 3 Nano, finds official guidance with Tavily, and writes a cited plan and a draft reply with Nemotron 3 Ultra.

The result is a result-first workspace, not a report buried below the fold. Due date, amount and issuer lead; the plan follows. Every step that depends on a procedure, fee, portal or deadline must cite a URL from the Tavily result set. Invented URLs are stripped on the server, and unsupported steps are marked “General advice · no source”. Sources are labelled as an official page, issuer page, source page or demo page, never a blanket verified check.

Deadlines export to an .ics calendar file; the plan exports to Markdown. The UI shows a real pipeline trace with model, latency and token usage.

How we built it

  • FastAPI backend and vanilla JavaScript frontend, no build step.
  • One OpenAI-compatible client on Nebius Token Factory, routed per task by model ID and per-role base URL. Vision, triage and reasoning run on three different NVIDIA Nemotron models behind the same request shape.
  • Strict JSON prompts with Pydantic validation, think-block stripping, a one-shot JSON retry, and a 4,000-token plan budget.
  • Tavily Python SDK with an issuer portal query, official-domain ranking, social and aggregator filtering, and issuer relevance filtering before the planner sees any source.
  • A server-side source whitelist so the planner cannot cite invented links.
  • Result-first two-pane UI: intake and document facts in a rail, the case file beside it, a visible calendar action, and facts, draft and sources in disclosures.
  • Pillow downscales images to 1600 px and re-encodes JPEG q85 before sending a base64 data URL.
  • .ics generated by hand per RFC 5545, with no calendar library.
  • Images are processed in memory and never written to disk.

Challenges

  • Reasoning-first defaults. Super-VL and Nano returned empty content with finish_reason="length" until we set chat_template_kwargs.enable_thinking=false for those two stages. Ultra keeps full reasoning.
  • Plan truncation. Ultra sometimes hit the 2,000-token ceiling and returned half a JSON object. The planner now has 4,000 tokens, and the failure is visible in the trace.
  • Source relevance. Search initially returned an unrelated US utility commission for a Bangalore bill. The search now queries the issuer portal directly and drops sources that do not mention the issuer.
  • Model and region routing. Models live in different regions, so each role has its own base URL and vision or reasoning failures retry once on the Super fallback.
  • UX. The first build put the result below a landing page. We rebuilt it around the result, with a fixed calendar action on mobile and focus moving to the result heading.

Accomplishments

  • A complete product flow: upload, read, verify, plan, reply and calendar export.
  • Official BESCOM pages cited for the sample bill, with unsupported steps clearly labelled.
  • A live trace that shows which NVIDIA model ran, how long it took and how many tokens it used.
  • Honest source tiers and a demo mode that labels canned data rather than pretending it is live.
  • 31 automated tests covering JSON extraction, source ranking, calendar and Markdown exports, API errors and the demo path.

What we learned

Right-sizing models per step matters. Nano handles triage in a fraction of the time and tokens that Ultra would take, leaving Ultra for the one step that genuinely needs reasoning. Source quality matters as much as model quality: a cited link is only useful if it actually supports the step.

What's next

  • Native Android capture inside Keepary.
  • Multi-page PDF support.
  • Scheduled reminders with Nebius Serverless Jobs.
  • Hindi and Kannada output.

Built With

Share this project:

Updates

Submission history