AI-shopkeeping for the Heartlands

We make shopkeeping less about administrative work; more about building customer relations, so that merchants spend less time doing repetitive work and instead focus on building the business.

Comment

The Problem: A Judgment-Bandwidth Crisis, Not a Chatbot Problem

Ah Kong is 51. For ten years the shutters of his provision shop have gone up before 7am and come down after 9pm — every day, by himself.

His day, on loop:

Time What he does What it costs him
6:45am Shutters up, restock the fridge, catch the office rush The one hour where footfall data would actually change a decision
1pm Quote the same three suppliers, same prices, same small talk 40 minutes of typing he cannot delegate
7pm Walk the shelves, mentally flag what's low, hope he remembers Reorder points held in working memory, lost by morning
10pm Shutters down. A bad Google review sits unanswered for six days Reputation decay he never sees until it compounds

He isn't struggling to survive. He's stuck in survival mode. Every hour spent quoting suppliers or counting stock in his head is an hour not spent on the one thing only he can do: reading a regular's mood, deciding which promotion is worth running, deciding what the shop should become next.

Five specific failures compound the problem for solo heartland operators:

# Challenge Why it persists
1 Hands are never free Serving customers means no typing, no clicking, no dashboards. Any tool requiring a keyboard is a tool that goes unused.
2 Language fragmentation Reviews and supplier messages arrive in English, Mandarin, Malay, Tamil, and Singlish. Replying well in each takes time he doesn't have.
3 Reputation runs unattended Google, Facebook, Instagram, and Reddit mentions accumulate with nobody watching. Problems surface as crises, not as trends.
4 Business memory lives in one head Supplier terms, reorder points, rainy-Friday sellers — none of it is written down, none of it survives a sick day.
5 Generic AI is unsafe here A model that auto-sends a message to a supplier, or auto-posts a reply to a bad review, is a liability, not an assistant.

A chatbot answers questions. Ah Kong doesn't have a question-answering problem.

The Solution: What Agnes Does

Capability Details
Hands-Free Operation Fully voice-activated console — merchant data, sentiment, inventory, protocols, all reachable by speaking
Drafts, Never Sends Every outbound message is staged into state['staged_actions'] and requires a second, explicit merchant approval; Google review replies are draft-only
Multilingual by Default Agents instructed to match the incoming language — Mandarin in, Mandarin out; Singlish is fair game
Cross-Platform Monitoring Sentiment aggregation over Google Reviews, Facebook, Instagram, and Reddit with navigable mention cards
Business Memory Per-tenant Firestore persistence with progressive extraction — facts are learned session by session, confidence-scored, and correctable
Confidence-Aware Hedging ≥0.85 stated plainly · 0.6–0.85 gets "(please confirm)" · <0.6 gets "(rough estimate — correct me if wrong)"
Crisis Protocols Health inspection, supply disruption, viral negative review, staff shortage, equipment breakdown — checklist and plan surfaced on voice command
Situational Awareness Screen share + shop photo capture streamed to Gemini so Agnes can see the dashboard or the shelf being discussed
Intelligent Routing One orchestrator, 5 specialist agents, 36 tools — the merchant just talks
Hallucination Prevention Argument whitelists before every tool call · render_command schema check after every response · prompt-injection containment on the merchant brief

Agnes listens continuously in the storefront. The merchant speaks naturally — "Agnes, check my customer reviews", "help me reply to the negative review about wait times", "what's my inventory status?" — and Agnes responds with the right card on the console and a brief spoken confirmation.

Agnes proposes and confirms. She does not act unilaterally. The merchant decides; Agnes makes sure they have what they need to decide well.

How we built it

Architecture

Architecture Diagram: https://github.com/lohjo/agnes-hlm/blob/main/Gemini-XPRIZE-Architecture-Diagram.png

System Overview

Browser (Merchant Console)
  │  16 kHz PCM audio (3200-byte chunks)  +  JPEG screen/shop frames
  │  ◄── 24 kHz PCM audio  +  JSON render_commands
  ▼
FastAPI WebSocket Server (Cloud Run)   /ws/{user_id}/{session_id}
  │  upstream_task()  browser → Vertex AI
  │  downstream_task() Vertex AI → browser
  │  LiveRequestQueue  →  ADK Runner  →  Vertex AI Live API
  │                                            │
  │              Gemini 2.5 Flash Native Audio Dialog
  │              ASR · LLM reasoning · Native TTS · Function calling
  ▼
hlm_Orchestrator (root_agent)
  ├── 26 direct tools  (merchant data · inventory · sentiment · docs · HECS · KampungConnect)
  └── 5 specialist sub-agents
        Message_Agent · Marketing_Agent · Trend_Engine
        Complication_Advisor · Visual_Assistant
  ▼
Firestore  tenants/{tenant_id}   — hydrate on connect, flush on close
  └── degrades to in-memory dev store when credentials are absent

Two frontends share the server: /console is the frozen vanilla-JS voice runtime; /login, /signup, /onboarding, /profile, /settings, /history are a React 19 + TanStack Router + Tailwind 4 island whose built bundle is committed under app/static/app/.

Agentic Workflow

Workflow Diagram: https://github.com/lohjo/agnes-hlm/blob/main/Gemini-XPRIZE-Workflow-Diagram.png

Challenges we ran into

Critical Thinking and Problem Solving

One challenge during the project was ensuring that ideas discussed in meetings were accurately captured and understood by everyone; records were inconsistent, so decisions relied on memory. Inspired by Working Backwards, we introduced a shared PR-FAQ (Fig. 5) narrative document to record meeting discussions, observations, evidence, assumptions, proposed solutions, and unresolved questions.

Rather than simply documenting outcomes, we used it to interrogate each idea's evidence and assumptions against our goals. This shifted our discussions from opinion-based debates to evidence-based decision-making, strengthening the quality of our analysis.

The document also became a single source of truth, allowing team members to stay aligned even if they missed meetings and reducing repeated discussions. This improved communication, collaboration, and confidence in our final recommendations.

Through this experience, we strengthened our critical thinking by learning to evaluate evidence, challenge assumptions, and communicate ideas clearly rather than simply generate solutions.

Accomplishments that we're proud of

It isn't the voice interface nor the multi-agent routing, but the memory layer and how it made the entire system operable by one person.

Heartland Copilot converts messy, half-said merchant conversations into structured, provenance-tracked business memory. Every fact the AI holds carries an extracted_session marker (when it was learned) and, where inferred rather than stated, a confidence score that triggers hedge-and-confirm behavior instead of flat assertion. The system's certainty is legible and correctable, not hidden inside a black-box context window.

One root orchestrator routing to specialist agents for messaging, marketing, trend analysis, crisis handling, and visual monitoring. Every tool call passes through a grounding layer that whitelists arguments before execution and validates structured render output afterward, closing off the two failure modes that make agentic systems unsafe for a real business: hallucinated actions and silent tool misuse.

A one-person company cannot staff separate teams for customer messaging, marketing copy, inventory monitoring, and crisis response. The multi-agent architecture does that staffing computationally. Each specialist agent handles a domain that would otherwise require a dedicated hire, while the orchestrator manages handoffs and priority the way a competent operations manager would. The memory layer is what makes this viable rather than chaotic: because every agent draws from the same structured, provenance-tracked business context, the one person operating the system doesn't need to brief each function separately or catch contradictions between them. The AI carries the institutional knowledge that, in a traditional business, lives distributed across employees' heads.

What we learned

Entrepreneurial Thinking

Entrepreneurship means solving problems people value.

We didn’t start with a solution, we started by understanding people in their own environments. We had assumptions about merchants' challenges but lacked evidence to justify them. Rather than jumping straight into solution development, we conduct a structured participatory research session during the site recce.

Merchants shared their workflows, frustrations and unmet needs instead of simply validating our assumptions, using statement starters to invite divergent thinking (LUMA, n.d.).

During the interviews, we documented recurring pain points, observed how merchants interacted with customers and compared responses across different businesses. This allowed our team to identify common patterns instead of relying on isolated opinions.

The insights collected became the evidence that guided our subsequent brainstorming sessions. Several initial ideas were discarded because they addressed symptoms rather than root causes. We evaluated new concepts based on user desirability, implementation feasibility and potential value to merchants before deciding which ideas to develop further.

Through this project, we learnt that innovation is driven by curiosity, validating assumptions through field research, and iterating based on real user needs.

What's next for AI-shopkeeping for the Heartlands

Current progress

  • Real and working: persistent per-tenant business memory (Firestore, confidence-hedged, progressive), session curation into structured summaries, Telegram integration with a hard human-approval gate before anything sends, Google Business Profile lookups, read-only web search, a React portal, 94 tests, and CI + a self-governing "loop" system (autonomous triage/PR bots that never auto-merge — auth/persistence stay human-only).
  • Named but not yet real: an "audit public presence" playbook, a true dynamic agent factory (today it's a fixed 5-agent roster + a template schema), browser actions (search-only so far), and WhatsApp (spec'd, no code yet).
  • Current friction: CI is red on main right now (a flaky mocked-API assertion), the daily "Product Designer Loop" action has failed ~6 days running, and 5 open backlog issues (WhatsApp module, contact directory, integration registry, two read-only UI surfaces) are unclaimed.
  • Strategic bet: "trace-grounded workflow memory" — reusable playbooks that improve from real merchant usage, built in a deliberately staged order (memory → integration registry → playbook runner → action queue with mandatory approval) rather than piling on integrations early.

Upcoming milestones

  • Two-sided thesis: the AI copilot buys back merchant time so they can meaningfully host a student; the student brings judgment/relationship-building the AI can't. One system closing the capability gap on both sides, not another tool layered onto a stretched merchant.
  • Proposed NP Business capstone structure: Phase 1 kickoff (bonding, Design Sojourn-facilitated design thinking, community partner insights, per-workshop reflection) → Phase 2 autonomous project work with merchants over semester break, periodic check-ins with a mentor → Phase 3 reflective final pitch scored on sustainability/feasibility/creativity/value (echoing HIC's feasibility/viability/desirability lens).
  • Format: non-compulsory, interest-based, non-graded, 1–3 months, kept selective/self-driven — students need real matched skills, merchants need real bandwidth and a genuine unmet objective.

AI Declaration

I/we have adopted, verified and adapted Generative Artificial Intelligence (Gen-AI) output for this submision.

AI Tool Used Prompt or Input to the AI How the AI Output Was Used
https://claude.ai/share/56501a21-fd98-448c-b508-dca426d9939f to find a research direction for project
https://claude.ai/share/56501a21-fd98-448c-b508-dca426d9939f summarising information about NP Business Studies (N45)
https://claude.ai/share/a3fcb113-9107-47c0-bf57-517100f6df45 summarising information about HECS
https://claude.ai/share/8a7df678-29de-4706-bb4d-7b7dc544ab5b Generating a cost model implementation using a proposal-format
https://chatgpt.com/c/6a65c3ad-9ec0-83ec-b844-6a784bf1855a how can i phrase this after students come down for their heartland heckathon with the merchants termly, they will gain valuable experience as to working with the merchants, hence getting a certificate of recognition for the 3 months programme whereby from the heartland pov, the students would have solve their root cause and provided relevant solution to aid in the problem. hence how i can say to make this heartland heckathon sustainable Gen Ai was used to paraphrase information about our implementation plan - pilot programme in bullet point format
https://chatgpt.com/share/6a662452-346c-83ec-b2a1-2fe6b70fe045 Gen AI was used to enhance the visuals of the slides. All the visuals and content suggested were reviewed before being used to reflect our work.
https://chatgpt.com/share/6a662542-87e4-83ec-acc6-e3f085a2edf0 Gen AI was used to improve the flow and wording of the presentation. All the content suggestions were reviewed and rephrased before being used to reflect our work.
https://claude.ai/code used to code the app infrastructure
https://claude.ai Generate me a cash flow statement year-by-year for the hackathon these are the requirements: infrastructure costs (initially to build it == 30 -- claude code + 30*0 -- cursor (won), education costs (i.e. NP design thinking workshops, HECS module training, mentoring), planning costs, marketing costs, maintenance costs (AI hosting + cloud deployment n storage) ==, costs bourne by XX, Total cost revenue generated from B2B subscriptions, grants (i.e. PDG, AIxDL, HEPG, SMEs go digital, enterprise SG, HECS & FMAS), total revenue from subscriptions + grants, total revenue from subscriptions, net cash remaining do these over a 6 year period with the following assumptions: for year 1, assume that we can garner 1000 users with its active user base being 20% of that. The app uses a freemium model -- rate limits on the free version, unlimited on the paid one (15/month -- still tentative -- need user retention, pilot testing, etc) but data shows that merchants wld pay 50 (2), 20 (1), 10(1) and 0 (1) with most still trying to understand use case (AI shopkeeping at the counter -- promising but narrow whitespace), data also shows that given a imaginary budget of 100 SGD the distribution is like: Feature

Built With

Share this project:

Updates