Inspiration
A small shop does not need another dashboard demanding constant attention. It needs the work behind the counter to get done.
Duka Autopilot explores that idea for a Kenyan neighbourhood shop: customers order in English or Kiswahili, sometimes by voice note; sales still appear in handwritten ledgers; and payment records do not always line up neatly with orders. The owner should not have to translate all of that into perfect prompts or spend the evening resolving every routine transaction.
Our central design principle is simple: exact evidence can move the work forward; uncertainty must remain visible to the owner.
What it does
Duka Autopilot brings order intake, ledger capture, reconciliation, and owner decisions into one back office.
- Morning Brief gives the owner a summary and the decisions needing attention.
- Ledger Desk turns a photographed handwritten page into proposed sales. Each row is validated independently: clear entries can be recorded while uncertain quantities or amounts go to review.
- Decisions shows the evidence, reason for stopping, and proposed effect. Approval is a meaningful boundary, not a decorative confirmation screen.
- Customer Inbox accepts text and voice-note orders, including Kiswahili. Gemini interprets the request; tools resolve products and calculate prices from the shop catalogue.
- Night Shift implements exact payment reconciliation first, bounded model-assisted review of the remaining ambiguity, restock proposals, and a persisted report. Cloud Scheduler is configured to launch nightly and morning-digest jobs.
This is a synthetic-data demonstration, not a live banking or WhatsApp integration. The application does not transfer money or place supplier orders. Refund and restock decisions support manual fulfilment.
How we built it
The owner console is a standalone Next.js application on Cloud Run. Its backend-for-frontend calls a private FastAPI service; Pub/Sub delivers asynchronous work to a private Cloud Run worker. Firestore stores business records, approval state, and durable control records. Cloud Scheduler invokes the background jobs.
Google ADK 2.7.1 orchestrates the workflow, with gemini-3.7-flash on Vertex AI interpreting text, audio, and images. Model-based classification and extraction sit alongside deterministic routing, exact reconciliation, and validation tools. The model cannot choose arbitrary catalogue prices or turn an ambiguous match into an authorised payment.
The cloud runner integrates managed Sessions and Memory Bank through the ADK Vertex AI services. Session state supports workflow continuity; memory is advisory context, never the source of authorisation or financial truth. Firestore remains authoritative. Configuration is verified in the deployed release; recovery and memory behaviour require their own end-to-end evidence.
Four interfaces separate storage, event delivery, sessions, and memory from domain logic. Local development uses SQLite and local adapters; cloud deployment uses Firestore, Pub/Sub, and managed context. GitHub Actions builds and smoke-tests images, and a manually triggered production workflow promotes the selected release images by digest.
Challenges we ran into
Moving from local state to managed sessions exposed a real integration challenge: application conversation identifiers and managed-session resource identifiers are not interchangeable. We added explicit session-ID handling and stale-pointer recovery. That experience reinforced why a passing health check is not enough to demonstrate a background agent.
Ledger extraction required a second boundary: recognising handwriting does not justify writing every recognised row into the books. Per-row validation lets a useful page proceed without silently guessing its doubtful entries.
We also separated model evaluation from deterministic testing. A passing business-rule test proves something different from a successful model-assisted conversation or an overnight cloud execution.
Accomplishments that we're proud of
The two deployments running on Cloud Run, the ADK workflow implemented and the integration with Gemini and Agent platform to build a solution that works for a typical African business
What we learned
An agent's usefulness is not proportional to how many decisions are delegated to a language model. Reliable tools, bounded work, and clear exception handling make the model more useful.
Multimodality also needs precise safety claims. Text screening does not prove that every instruction hidden in audio or an image will be detected. Shared validation and authorisation boundaries remain essential after interpretation.
What's next for Duka Autopilot
Complete release-specific overnight, recovery, and memory verification; pilot with consenting shop owners; add a real WhatsApp Business channel and authorised payment-data integration; and validate the workflow against real operating needs before handling live customer or financial data.
Log in or sign up for Devpost to join the conversation.