DocSaathi

A private, local AI assistant that reads a photo of an English document and explains it in Tamil, Hindi or simple English: what it is, what to do, by when, and how much. Nothing leaves your machine.

Comment

Inspiration

Much of everyday Indian life runs on English paperwork: electricity bills, bank notices, insurance letters, rental agreements, tax and legal notices. Many people who must act on these read only Tamil, Hindi or another regional language. So they ask a shopkeeper, neighbour, agent or cyber-cafe operator to read the document, and often hand over their Aadhaar, PAN and account numbers while they do. Helpers can misread, overcharge or commit fraud. Deadlines get missed, and fake "your account will be blocked" notices are hard to tell from real ones.

Cloud AI and camera-translate apps mean uploading private documents to someone else's servers, and translation alone doesn't answer the real questions: what do I do, by when, how much, and is this letter genuine?

Translation tells you what the words say. DocSaathi tells you what to do.

What it does

DocSaathi is a private AI assistant that runs entirely on one laptop. You give it a photo of an English document, and it:

  • Reads the document with Gemma 4 vision, running locally through Ollama.
  • Extracts the key facts: document type, sender, amount due, due date with days left, reference IDs, required actions and penalties.
  • Explains it in plain language in Tamil, Hindi or simple English, streamed as it is written, leading with what to do, by when and how much.
  • Shows its evidence. Every fact comes with the exact words it was read from, next to the transcription.
  • Answers follow-up questions from that document only. If the answer isn't in the document, it says so and suggests who to ask.
  • Protects privacy by redacting Aadhaar, PAN, phone and account numbers.
  • Flags scam signals with a verdict of likely genuine, suspicious or likely scam.
  • Creates a calendar reminder (.ics file) for the due date.

Nothing leaves the machine, so nobody has to hand their Aadhaar to a stranger again.

How we built it

The core design rule is code handles facts, the model handles language. Language models misread numbers and invent dates, so we don't let the model decide facts.

  1. Pass 1, understand. Gemma 4 transcribes the image and returns structured JSON, with a quoted source_text for every fact.
  2. Verify, in plain code. A deterministic engine checks that each date is valid and plausible, calculates days left and overdue status in UTC, confirms the amount appears in its quote, and confirms each quote really exists in the transcription. Matching is Unicode-safe, so Tamil and Devanagari text work. Anything doubtful sets a needs_paper_check flag, and the app tells the user: "Please check this against your paper copy."
  3. Pass 2, explain. Gemma receives only the verified facts, with no image, and writes the explanation in the chosen language.
  4. Grounded Q&A. Each answer ends with a source quote, and the server locates it in the transcription. If it can't be found, nothing is highlighted, so the app never points at the wrong place.

The scam check combines Gemma's judgement with rules (OTP requests, UPI requests, urgency wording, short links). Rules can raise the verdict but never lower it.

Privacy is enforced in code, not just promised:

  • The server refuses to start unless the Ollama address is localhost, 127.0.0.1 or ::1.
  • It listens on 127.0.0.1 only, and CORS allows only localhost origins.
  • Request bodies, images, transcriptions and model output are never logged, only error codes.
  • Nothing is stored on the server.

Stack: React, Vite, Node.js, Express, Server-Sent Events, Ollama and Gemma 4. The backend uses only three dependencies (express, cors, dotenv), with no database, vector store or cloud service. It has 69 passing automated tests.

Where Gemma 4 is used: reading a phone photo, classifying the document and extracting facts with source quotes, writing Tamil, Hindi and simple English, answering questions from the document, and judging whether a letter looks like a scam. Plain code does all date, amount and source verification, redaction and calendar generation.

Challenges we ran into

  • Making a small local model reliable. Structured output and image input can conflict on some Ollama versions, so we added a prompt-only fallback with tolerant JSON parsing and one retry.
  • Exact quotes. Models paraphrase instead of copying. We matched quotes ignoring case, spacing and punctuation across scripts, and flag anything that can't be located.
  • Dates and time zones. Naive date maths gives the wrong "days left" near midnight, so all date logic uses UTC components with a testable clock.
  • Keeping private data out of logs. We audited every log line and error message so that no document content can leak.
  • Speed on a laptop. We used model warm-up, request timeouts, and a smaller model fallback.
  • Integrating four people's work in one day. We agreed an API contract early and fixed mismatches in port, language codes and response shapes.

Accomplishments that we're proud of

  • A working end-to-end flow where the numbers shown to the user are checked by code, not trusted from the model.
  • Privacy guarantees that are checked at startup rather than promised in a README.
  • Every fact in the UI can be traced back to the exact words in the document.
  • A backend with 69 passing tests and a documented API, built with small, steady commits across the day.

What we learned

  • Splitting the work so that code handles facts and the model handles language made the product both more trustworthy and easier to test.
  • For people who can't read the document, saying "I don't know" honestly matters more than sounding clever.
  • Open weights aren't a gimmick here. The privacy promise is only possible because the model runs locally.

What's next for DocSaathi

  • More document types and more Indian languages.
  • A form-filling guide for scheme applications.
  • Measured accuracy on a larger set of fake sample documents, including native-speaker review of Tamil and Hindi output.
  • Multi-page PDFs and offline voice read-aloud.
  • A shared-laptop mode for panchayat offices, CSCs and NGO desks.

Limitations

DocSaathi is a reading aid, not legal or financial advice. Accuracy depends on photo quality and the model. The scam check is a signal, not a guarantee. Serious matters should be confirmed with a bank, a CSC or a lawyer.

Built With

Share this project:

Updates