Inspiration

What it does

How we built it

Challenges we ran into

Accomplishments that we're proud of

About the Project

Inspiration

Companies can process hundreds or thousands of invoices, but extracting text from them is only the beginning of the problem.

The real challenge starts when someone needs to find, compare, aggregate, or reason across those documents. Traditional OCR can extract text, SQL can analyze structured data, and conventional RAG can retrieve semantically relevant documents, but each approach solves only part of the problem.

This inspired me to build Agent Ledger: an AI-powered invoice intelligence system that transforms invoice images into structured, searchable, and traceable financial knowledge that users can explore simply by asking questions in natural language.

The goal was to create something more than an OCR application or a basic RAG demo: a system capable of combining multimodal document understanding, semantic retrieval, deterministic analytics, and agentic reasoning behind a single conversational interface.

---

What It Does

Agent Ledger allows users to analyze large collections of invoices and ask questions about them using natural language.

For example, a user can ask:

Which supplier has the highest total invoiced amount across all invoices? Show the invoices used in the calculation.

Agent Ledger can analyze the complete collection, calculate totals while keeping different currencies separate, and return the individual invoices used as evidence, including their dates, amounts, currencies, and original source files.

Users can also ask detailed questions about individual invoices. In this case, Agent Ledger can use semantic retrieval to find the relevant document and provide richer information such as line items, supplier and buyer information, payment details, financial totals, and other information extracted from the original document.

An important principle throughout the project is traceability. Answers remain connected to the original invoice images, allowing users to verify the information against its source.

This also enables Agent Ledger to surface inconsistencies found in the original documents without silently correcting them. For example, if an invoice contains a due date that is chronologically inconsistent with its invoice date, the system can warn the user while preserving the information exactly as supported by the original document.

---

How I Built It

Agent Ledger is built around the Google Cloud and Gemini ecosystem.

The original invoice images are initially stored in Google Cloud Storage buckets, providing a centralized and scalable storage layer for the source documents before they are processed by the AI pipeline.

Gemini through Vertex AI performs multimodal document understanding directly from these invoice images. Instead of depending on a traditional OCR-only pipeline, Gemini extracts, understands, and structures the relevant financial information from each document.

The resulting structured invoice information is stored in Cloud Firestore, together with embeddings that enable semantic vector search.

The conversational and agentic layer is built using Google Agent Development Kit (ADK). Gemini interprets the user's natural-language request and determines which retrieval or analytical capability should handle it.

This creates a hybrid architecture:

• Google Cloud Storage stores the original invoice images.
• Gemini through Vertex AI performs multimodal document understanding and structured extraction.
• Cloud Firestore stores structured invoice data and embeddings for semantic vector search.
• Semantic retrieval handles questions requiring detailed or unstructured information from specific invoices.
• Deterministic analytics handles operations such as filtering, grouping, SUM, COUNT, MIN, MAX, and comparisons across the complete invoice collection.
• Google ADK and Gemini understand user intent, select the appropriate tools, and explain the results in natural language.
• Original invoice references are preserved as supporting evidence, maintaining traceability between answers and their source documents.

Finally, the application is deployed on Google Cloud Run, providing a cloud-hosted interface to the complete system.

The overall flow can be summarized as:

Cloud Storage → Gemini / Vertex AI → Firestore + Vector Embeddings → Google ADK → Semantic Retrieval or Deterministic Analytics → Grounded Answer + Evidence

---

Challenges I Faced

One of the most important challenges appeared when scaling analytical questions across the complete invoice collection.

Initially, global questions required retrieving all invoice metadata and allowing the language model to perform grouping, currency normalization, calculations, and response generation itself.

This worked, but it introduced significant latency and used the language model for operations that did not actually require AI reasoning.

The key realization was that financial arithmetic is deterministic and should not be performed by a language model when conventional code can perform it faster and more reliably.

I redesigned this part of Agent Ledger so that deterministic operations are executed directly by the application. Python performs calculations, grouping, filtering, currency normalization, and prepares the supporting evidence. Gemini can then concentrate on understanding the user's request and explaining the already-computed results.

This dramatically improved response time while also making financial calculations more reliable.

Another challenge was handling different representations of currencies. For example, $ and USD may represent the same currency. Agent Ledger normalizes these values deterministically before aggregation while ensuring that genuinely different currencies, such as USD and EUR, are never incorrectly combined into a single monetary total.

A further challenge was deciding when semantic retrieval was appropriate.

A conventional RAG approach based on top-k similarity is useful when looking for specific information, but it can produce incomplete results for questions that require the entire dataset. Agent Ledger therefore uses different strategies depending on the intent of the question rather than treating every request as the same retrieval problem.

---

What I Learned

Building Agent Ledger reinforced one of the most important lessons I learned during this project:

Not every problem inside an AI application should be solved by AI.

Large language models are extremely powerful for understanding intent, reasoning about unstructured information, semantic retrieval, and explaining results.

Traditional deterministic code is considerably better for exact arithmetic, filtering, grouping, counting, sorting, and aggregation.

Combining both approaches produced a system that is faster, more reliable, and easier to audit.

I also learned that RAG does not need to mean simply retrieving the top k most similar documents.

A question about a specific invoice may require semantic vector retrieval, while a global financial question may require operations across the complete dataset.

The agent should be able to understand this difference and select the appropriate strategy.

This led to one of the core design principles behind Agent Ledger:

Use AI where reasoning is required, and deterministic computation where exactness is required.

---

Why Agent Ledger Is Different

Agent Ledger combines the flexibility of natural language and semantic RAG with the reliability of deterministic data operations.

Users do not need to know SQL, understand the underlying database schema, manually construct filters, or search through large collections of invoices themselves.

Instead of writing queries involving WHERE, GROUP BY, SUM, COUNT, or date operations, users can simply describe what they want in natural language.

The model also does not need to generate and execute arbitrary SQL statements from user input. Analytical capabilities can instead be exposed through controlled and validated application tools, reducing the attack surface associated with arbitrary AI-generated SQL and SQL injection.

Agent Ledger also addresses an important limitation of conventional RAG systems. A semantic top-k search may be excellent for finding relevant documents, but it cannot guarantee that the retrieved subset represents the complete dataset required for a global financial calculation.

Agent Ledger can choose between semantic retrieval and complete deterministic analysis depending on the question.

At the same time, users do not simply receive database rows or numerical results. Gemini explains those results in natural language, making complex financial information easier to understand.

Most importantly, Agent Ledger does not just provide an answer.

It provides the evidence behind that answer.

Invoice IDs, dates, amounts, currencies, suppliers, and original source references can remain attached to the result, allowing users to understand not only what Agent Ledger concluded, but also where the information came from.

The result is a system where large collections of invoices become financial knowledge that users can search, analyze, understand, and verify through natural language.

What we learned

What's next for Agent Ledger

Built With

• adk
• firestore
• google-adk
• google-cloud
• python