Khata AI

Inspiration

A typical Bangladeshi mudi dokan has 20–30% of its working capital trapped in baki — credit written by hand in a paper notebook. The shopkeeper can't see his total receivables, can't tell who's been overdue for three months, and feels awkward asking a neighbour, so small debts quietly become permanent losses.

Digital ledger apps exist, but they demand typing every entry — which a shopkeeper serving customers all day simply won't do. And the notebook isn't only storage. Writing the amount in front of the customer is what makes the debt socially binding. Replace the notebook and you break that trust.

What it does

Khata AI reads the paper khata instead of replacing it.

The shopkeeper keeps writing exactly as before, then photographs the page.

• Extract — Gemini 2.5 Flash reads the handwritten Bangla into structured entries: name, item, amount, date — with a confidence score on every amount.
• Review — rows below 0.8 confidence are flagged amber. He corrects three entries instead of typing forty, and nothing saves until he confirms.
• Dashboard — total outstanding, debtors sorted by most overdue, aging buckets of 0–30 / 31–60 / 60+ days.
• Listen — many shopkeepers are busy behind the counter or don't read comfortably, so খাতা শুনুন has Gemini write a spoken Bangla summary and ElevenLabs read it aloud.
• Remind — Gemini drafts a short, warm Bangla message for an overdue customer. The shopkeeper edits and sends it himself. The app never messages anyone automatically.

How we built it

Next.js 16 App Router, TypeScript, Tailwind v4. Four server-side routes:

• /api/extract — Gemini vision with a forced JSON responseSchema
• /api/summary — conversational Bangla, numbers spelled out for speech
• /api/remind — warm, informal Bangla reminder
• /api/speak — ElevenLabs eleven_v3, auto-falling back to eleven_multilingual_v2

All API keys stay server-side. No database — the ledger lives in localStorage, so the shop's book never leaves the phone. Shipped as a Docker image behind an nginx reverse proxy.

Challenges we ran into

Calibrated confidence was the hard part. Gemini initially returned 1.0 on every single row — including ones it had misread. A confidence score that's always 1.0 is worse than none, because it teaches the shopkeeper to trust everything.

We fixed it by naming the failure mode in the system instruction: the digit pairs that actually confuse in Bangla handwriting (১/৭, ২/৩, ৪/৮, ৫/৬, ৬/৯), plus temperature: 0 and a thinking budget for reasoning over ambiguous strokes. Confidence became meaningful.

Bangla-Indic numerals needed explicit mapping (৪৫০ → 450). We also hardened the Gemini client with retry-and-backoff on 429/5xx, abort timeouts, and finishReason checks so a safety block or token limit surfaces as a real message instead of a crash.

Accomplishments that we're proud of

The extractor correctly skips crossed-out rows — a paid debt struck through with a pen line is ignored, which is exactly what a shopkeeper expects. And the whole interface is in Bangla, down to the numerals: ৳৫,৮৩০, not ৳5,830.

What we learned

The review screen matters more than the extraction. Gemini still occasionally misreads a digit at high confidence. The product's job isn't to be perfect — it's to make the one wrong row easy to catch before it's saved. Designing for the model being wrong turned out to be the actual design problem.

What's next for Khata AI

Fuzzy name matching so করিম মিয়া and Karim Mia merge into one debtor, and prompting for the year when handwritten dates don't carry one.

Built With

• docker
• elevenlabs
• gemini-2.5-flash
• google-gemini
• next.js
• node.js
• react
• tailwindcss
• typescript