Aisle Whisper

• 

  Aisle Whisper Logo

• 

  Aisle Whisper - Software Architecture

Inspiration

Retail shelf checks are still highly manual in many stores. Associates walk aisle by aisle, visually scan products, and then try to remember what to restock or fix. We wanted to reduce that friction with a mobile-first AI workflow that works in real time: point the camera at the shelf, detect actionable issues, and generate a checklist immediately.

We focused on common retail pain points:

• Shelf checks are still manual and time-consuming.
• Issue identification is often inconsistent across team members.
• Teams need actionable outputs, not just AI observations.

What it does

Aisle Whisper is our fullstack shelf-audit MVP built for the Gemini Live Agent Challenge. It combines a PWA frontend with a cloud backend to deliver live overlay detections for common shelf issues (GAP, MISALIGNED, LOW_STOCK, OUT_OF_PLACE) and a structured checklist that teams can act on right away.

How we built it

We built Aisle Whisper as a cloud-hosted, mobile-first architecture:

• Frontend PWA: React + TypeScript + Vite + MUI
• Backend: Node.js + Express + WebSocket relay
• Hosting: Firebase Hosting (frontend) + Cloud Run (backend)
• AI integration:
• Gemini Live API for realtime session handling
• Google GenAI SDK for strict frame-based detection and structured output

Challenges we ran into

The biggest challenge was making realtime behavior reliable on real mobile usage instead of demo-only conditions.

Main issues we had to solve:

• Live websocket session stability and model/session setup differences.
• Parsing variability from model responses into strict overlay JSON.
• Preventing detection drift when users switch focus modes.
• Handling stale overlay updates and synchronization timing.
• Keeping the UX compact and clear on iPhone screen constraints.

Accomplishments that we're proud of

One accomplishment we are especially proud of is turning a complex real-time AI concept into a dependable, mobile-ready workflow that actually helps with day-to-day retail operations. We successfully integrated Gemini-powered multimodal analysis into a fullstack product on Google Cloud, delivered strict focus-based detection modes to reduce noise, and connected live overlays to actionable checklist generation with frozen visual context for clear handoff. Beyond features, we are proud of the engineering discipline behind the MVP: reducing scope when needed, hardening reliability on real devices, and shipping a coherent experience that demonstrates practical value instead of just a prototype demo.

What we learned

This project taught us important lessons about production-minded multimodal agents:

• Reliability beats feature count in live demos and real operations.
• Strict contracts and backend enforcement are essential to reduce hallucinated or off-scope labels.
• Mobile UX details matter as much as model quality.
• A clear operational output (checklist) increases product value more than raw detections alone.
• Fast feedback loops with real-device testing are critical for realtime systems.

What's next for Aisle Whisper

Next for Aisle Whisper is evolving from a strong MVP into a production-ready retail operations assistant with deeper workflow integration and measurable impact. Our roadmap includes reconnecting robust voice interaction with interruption handling, adding role-aware task assignment and escalation, integrating with inventory and planogram systems for closed-loop execution, and expanding analytics to track shelf-health trends, compliance, and replenishment performance over time. We also plan to strengthen deployment automation, observability, and multi-store configuration so teams can scale from pilot aisles to chain-wide adoption while keeping the mobile experience fast, reliable, and intuitive.

Built With

• express.js
• firebase-hosting
• genai
• google-cloud
• local-storage
• mui
• node.js
• pwa
• react
• rest
• typescript
• websockets