Zaptory

A smart middleware that converts product scans into structured inbound inventory for ERP/POS systems, automating invoice reconciliation and expiration date tracking.

Comment

Inspiration

It all started with a real-world need expressed by a friend of mine. He works as a warehouse manager at a quality measurement center in Vietnam. Managing a vast array of laboratory chemicals required a tool capable of converting images into data tables with high accuracy, as well as a simple system to track expiration dates. Based on his requirements, I developed and expanded the solution into Zaptory.

What it does

  • Zaptory is an AI middleware that enables businesses to digitize their inventory into data tables without replacing their existing POS or ERP systems. Users simply take a photo of a product label or an incoming shipment invoice; the system utilizes Gemini to automatically extract details such as product names, quantities, expiration dates, and batch numbers. It then generates a file ready for direct import into platforms like KiotViet, Sapo, Shopify, or Wix—eliminating the need for manual data entry.

  • Under the hood, the system employs a multi-agent architecture running on Google ADK 2.x. An invoice reconciliation agent matches scanned data against the actual invoice; if a significant discrepancy is detected, the process pauses to await user confirmation before proceeding. This state is persisted, ensuring that even if the backend restarts mid-process, the workflow resumes exactly where it left off without losing progress. Concurrently, other agents monitor expiration dates to issue alerts, detect duplicate images, and compile daily reports for warehouse managers.

How we built it

  • The backend is built on Google ADK 2.x using graph-based workflows rather than autonomous agents freely calling tools. Each processing step functions as an independent node with clear routing thresholds, allowing the logic to remain deterministic without relying on the LLM for every step. Gemini (via Vertex AI) is invoked only when genuine reasoning is required—such as matching ambiguous SKUs or analyzing duplicate images. The entire system runs on Cloud Run, with data stored in Firestore. Regarding the human-in-the-loop mechanism: upon detecting data discrepancies, the workflow had to truly pause (utilizing the ADK's native RequestInput to halt execution and checkpoint the state) rather than simply setting a dummy status variable and calling it a day. We then built an endpoint for user approval, allowing the agent to resume precisely where it left off—even after a backend restart.

  • The frontend is a Flutter app (compatible with both mobile and web) featuring a two-tier image deduplication process: it performs a quick local filter first, sending only ambiguous cases to Gemini Vision to minimize costs and latency.

Disclosure: The Flutter frontend app (scanning UI, Excel export) predates this hackathon under the name Scan2Sheet. All Google ADK 2.x agent orchestration, the multi-agent backend, Cloud Run deployment, and the Human-in-the-Loop reconciliation system — the core technical contribution of this submission — were built entirely during the Submission Period (Aug 3–31, 2026).

Challenges we ran into

  • The biggest hurdle was correctly implementing the human-in-the-loop mechanism for ADK 2.x. Our initial attempt merely set a "pending" status field and returned; while it appeared to be awaiting approval, the process had actually terminated, leaving nothing truly waiting. We had to delve deep into the ADK's native RequestInput mechanism to ensure the workflow genuinely paused and auto-checkpointed its state, then build a proper resume endpoint to continue from the exact point of interruption.

  • Next was the issue of session persistence on Cloud Run. We initially planned to use local SQLite to store state, assuming it would survive restarts. However, Cloud Run uses an ephemeral filesystem; if the container instance was replaced (due to a crash, scaling to zero, or a new deployment), all SQLite data would be lost. We had to recognize this limitation before accidentally pushing to production and explicitly acknowledge it as a deliberate trade-off for the demo, rather than a final solution.

  • Finally, there was the challenge of clearly distinguishing between "tested via mocks" and "actually working in production." We often thought a feature had passed all tests, only to realize the testing was limited to the service layer—never traversing the actual user flow or involving a manual visual inspection of the output files. Conducting multiple rounds of audits—insisting on concrete evidence (actual logs and files) rather than relying on "done" reports—helped us catch critical bugs before submission; for instance, an export template had a column mapping error that initially appeared fine.

Accomplishments we're proud of

  • Built a genuine "human-in-the-loop" mechanism—not a simulated pause—where the workflow halts, awaits approval, and resumes exactly where it left off, even after a backend restart. Verified this on actual Cloud Run, not just in a local environment.

  • Migrated all 5+ agents to the ADK 2.x graph workflow architecture, clearly separating deterministic logic (zero-token usage) from LLM-dependent tasks, thereby reducing both costs and latency.

  • Self-detected and fixed multiple critical issues before submission—ranging from fake HITL implementations and sessions failing to survive Cloud Run restarts to secret leaks—by consistently demanding actual evidence rather than accepting "done" reports.

  • Most importantly, we created a product that solves a real-world user problem.

What we learned

  • Gain a deeper understanding of the ADK 2.x graph workflow—specifically how to decouple deterministic logic (such as threshold routing or fuzzy matching) from nodes requiring an LLM to reduce token usage and latency, rather than routing everything through the model.

  • The ADK's "human-in-the-loop" mechanism involves more than just setting a "pending" status; you must use the correct RequestInput to truly suspend the workflow, automatically checkpoint the state to the session store, and resume at the exact node where execution paused once the user provides a FunctionResponse.

  • Cloud Run utilizes an ephemeral filesystem; sessions stored in a local SQLite database will be lost if the container instance is replaced (due to a crash, scaling to zero, or a new revision deployment)—a scenario vastly different from simply restarting a process on a local machine. This is a genuine limitation of serverless architecture that must be considered when designing state persistence.

  • Vertex AI and Google AI Studio employ different authentication mechanisms (service account/ADC versus API keys); it is crucial to select the appropriate method for a production Cloud Run environment rather than relying on personal keys used for testing.

  • Secrets should be managed via Secret Manager and injected using --set-secrets rather than being passed directly via --set-env-vars in the command line, as values ​​can easily be exposed in logs, command history, or revision metadata if not handled carefully.

What's next for Zaptory

  • Finalize and fully validate the mappings for export templates across various platforms (KiotViet, Sapo, Shopify, Wix)—while the Generic CSV format is already stable, the remaining templates require further verification before moving to production.

  • Add features for periodic inventory checks and direct camera scanning (real-time scanning) instead of relying on discrete photos, enabling a faster and more seamless inventory process.

Built With

Share this project:

Updates

Submission history