-
-
main page (upload page)
-
main page (upload page)
-
processing page for Review
-
Review page
-
Review page
-
Review page
-
Review page
-
Review page
-
Exported record
-
Exported record
LedgerPilot — Turning Handwritten Business Records into Financial Documents
Inspiration
The inspiration for LedgerPilot came from a real-life observation. My mother runs a small business, and like many local shopkeepers in Maharashtra, she maintains business transactions in handwritten notebooks commonly known as Bahi-Khata.
These notebooks contain valuable information about daily sales, services, and business earnings. However, organizing this information into structured financial documents can be difficult and time-consuming.
The challenge becomes even greater when records are handwritten in Marathi, the local language used by many shopkeepers. Most digital accounting workflows expect structured data or are primarily designed around English, creating an additional barrier for people who maintain their records in their own language.
This made me think:
Why should small business owners have to change the way they maintain their records just to make their financial information easier to organize?
I decided to explore this problem by building LedgerPilot and using actual handwritten business records from my mother's shop as a real-world use case.
What We Built
LedgerPilot is an AI-powered financial documentation tool that converts handwritten business ledger entries into structured sales records.
A user uploads a photograph of their handwritten ledger. The system uses AI to extract transaction descriptions, dates, and amounts, then organizes the information into a structured format.
Users can review uncertain entries, correct descriptions or amounts, and delete incorrect entries before generating their final document.
LedgerPilot also performs arithmetic checks to identify discrepancies between transaction amounts and recorded subtotals.
After reviewing the records, users can generate a professionally formatted Business Sales Statement PDF.
Our goal is to reduce the manual effort involved in organizing handwritten business records and make financial information easier to understand and present as supporting documentation.
How We Built It
We built LedgerPilot using a combination of frontend, backend, and AI technologies.
Technology Stack:
- Frontend: React, TypeScript, and Vite.
- Backend: Python and FastAPI.
- AI: Google's Gemini API for handwritten ledger interpretation and transaction extraction.
- Data Validation: Pydantic schemas and backend validation.
- PDF Generation: ReportLab for professionally formatted financial documents.
The workflow has five main steps:
- Upload: The user uploads a photograph of a handwritten business ledger.
- Extract: Gemini analyzes the image and extracts transaction information.
- Review: The user can inspect, correct, or delete entries, particularly where handwriting or amounts are unclear.
- Validate: The system recalculates totals and checks for arithmetic discrepancies.
- Generate: The reviewed records are organized into a downloadable Business Sales Statement PDF.
We focused on keeping the process straightforward so users would not need to manually recreate their entire ledger in a spreadsheet.
Building with Real-World Marathi Records
One important part of this project was testing it with actual business records instead of relying only on generated sample data.
I used handwritten records from my mother's shop, including Marathi transaction descriptions and handwritten amounts, to develop and test LedgerPilot.
I wanted to explore whether AI could help convert business records written in a local language into structured financial information.
English extraction is currently better implemented in the project. Marathi handwriting recognition is still an area that needs improvement. Handwriting variations, regional writing styles, abbreviations, and handwritten numbers make local-language extraction more challenging.
Working with these records helped us understand that supporting a language is not simply about translating text. Recognizing real handwritten information accurately is a separate technical challenge.
Challenges We Faced
1. Understanding Handwritten Records
Handwritten ledgers are not standardized. Different people use different handwriting styles, abbreviations, formats, and ways of recording amounts.
A small difference between two handwritten digits can change a transaction's value. Extracting text alone was not enough; we needed a process that allowed users to review and correct uncertain information.
2. Maintaining Financial Accuracy
One of our biggest challenges was ensuring that the system did not silently change amounts to make totals match.
We separated extracted transaction values from user-corrected values and implemented arithmetic checks so discrepancies could be identified instead of hidden.
We also needed to handle missing or unclear amounts without incorrectly treating them as zero.
3. Improving Local-Language Recognition
Working with Marathi records exposed limitations in handwriting recognition that were not as obvious when testing with English data.
We learned that real-world regional-language support requires more than a basic OCR or text extraction pipeline. It requires careful handling of handwriting, numbers, and context.
4. Designing a Simple Review Process
We wanted AI to reduce manual work, not create more work.
Making users confirm every transaction would defeat that purpose. We therefore worked toward automatically accepting sufficiently confident extractions while highlighting entries that genuinely need attention. Users can still edit or delete any transaction.
5. Working Within API Limitations
During development, we encountered Gemini API free-tier request limits. This encouraged us to think more carefully about when AI was actually necessary.
We focused on keeping calculations, editing, deletion, and PDF generation within our own application rather than making unnecessary AI requests.
What We Learned
Building LedgerPilot taught us that applying AI to a real-world problem involves much more than connecting an API to an interface.
We learned that:
- Real-world handwritten data is much less predictable than clean sample data.
- Local-language support is important when building technology for small businesses.
- AI extraction must be combined with validation and human oversight.
- Financial applications should preserve original information and avoid making assumptions.
- Missing information should never be replaced with invented values.
- A useful product should reduce the user's workload rather than introduce unnecessary steps.
Most importantly, we learned that technology should adapt to people's existing ways of working instead of expecting people to adapt completely to technology.
What's Next
LedgerPilot is a starting point toward making financial record organization more accessible to small businesses.
Future improvements could include:
- Better Marathi handwriting recognition and support for additional regional languages.
- Improved extraction accuracy across different handwriting styles.
- Reusable business profiles.
- Additional financial document formats.
- More robust validation and reporting features.
Our long-term vision is simple:
Help small businesses turn the handwritten records they already maintain into organized financial documentation, with less manual effort.
Built With
- artificial-intelligence
- computer-vision
- data-extraction
- data-validation
- edtech
- fastapi
- financial-technology
- generative-ai
- google-gemini
- handwriting-recognition
- image-processing
- large-language-models
- marathi
- natural-language-processing
- ocr
- pdf-generation
- pydantic
- python
- react
- reportlab
- rest-api
- typescript
- vite
- web-application
