Shopexa Ops: Proactive Ambient

Shopexa Ops turns Alexa+ into proactive procurement intelligence, detecting supply issues, spotting better prices, and staging purchase orders before you even ask.

Comment

💡 Inspiration

Current smart displays and AI assistants suffer from two fundamental limitations:

  1. They are strictly reactive: They sit idle, waiting for a human to ask them a question. But real-world business operations don't wait. If a supplier raises prices by 26%, or critical inventory runs out, an enterprise operator shouldn't have to manually prompt their smart display to discover it.
  2. They have no conversational memory across interruptions: In a busy office, ambient interruptions happen constantly. When an assistant interrupts with an alert while you're in the middle of staging an order, standard bots wipe your state, forcing you to start over. Furthermore, conventional smart assistants don't know who you are talking to—triggering accidental orders when you are simply presenting to colleagues or judges.

We were inspired by Amazon's vision for Alexa+ as an ambient, proactive intelligence. We built Shopexa Ops: an autonomous procurement agent that operates in the background of your business, monitors supplier changes, evaluates cost-effective alternatives, interrupts only when it has a verified solution ready to authorize, and maintains an Interruption Stack that safely suspends and resumes your tasks across conversational detours.

⚙️ What it does

Shopexa Ops is a voice-first, ambient procurement agent designed for small businesses, retail back-offices, and enterprise smart displays:

  • Proactive Ambient Intelligence: Simulates real-time supplier webhook events (e.g., RoastCraft raising Ethiopian coffee prices by 26%). Unprompted, Shopexa Ops wakes the display, delivers a voice announcement directly through Amazon Nova Sonic's native duplex audio stream, and displays a responsive 2x2 alternative comparison grid (Option 1: Colombian Supremo saving $11, Option 2: Kenya AA Reserve saving $7, or Keep Original).
  • Interaction State Machine & Interruption Stack (Suspend / Resume): Every interaction lifecycle is tracked (IDLE, ORDER_STAGED, ALTERNATIVE_SELECTION, BRIEFING_ACTIVE, PROACTIVE_INTERRUPT, SIDE_QUERY). If an ambient price spike directly matches the active product in focus, your active task is snapshotted and pushed onto an Interruption Stack (unrelated alerts are held in the Notice Tray to avoid conversational disruption). A topbar badge (1 Suspended (COFFEE)) appears. Saying "Back to my order", clicking "Dismiss", or letting the alert auto-dismiss automatically pops your task and plays a natural voice bridge: "Returning to your staged coffee order totaling $33.00. Ready to authorize."
  • Tiered Dual-Model Intelligence via AWS Bedrock:
    • Amazon Nova Sonic (amazon.nova-2-5-sonic): Powers ultra-low latency, full-duplex bidirectional audio streaming (16kHz PCM in → 24kHz LPCM out) with hardware-level barge-in interruption detection.
    • Amazon Nova Pro (amazon.nova-pro-v1:0): Handles deep autonomous planning, inventory queries, supplier pricing analysis, and Model Context Protocol (MCP) tool execution in under 800ms.
  • Smart Attention & Addressing Gate (attention_gate.py): Real-time syntactic classifier that identifies whether speech is addressing Alexa directly (DIRECT_COMMAND), narrating/presenting to hackathon judges (JUDGE_NARRATION), or having casual side-chatter (BYSTANDER_CHATTER). When not addressed, the assistant stays in attentive standby and suppresses sensitive purchasing tools.
  • Live MCP Telemetry HUD: Transparently renders every duplex audio event, attention classification, model handoff, and raw JSON-RPC tool call in real time.
  • AWS IAM Role Authorization: Enforces credential security before any purchase order is dispatched.

🛠️ How we built it

We completely broke away from the standard "chatbot" architecture to deliver a true Agentic Smart Display:

  1. Self-Hosted MCP Server (Spec 2025-11-25+, Streamable HTTP): Implemented in mcp_server.py and mounted onto FastAPI via streamable_http_app() at /mcp with fallback SSE support. Exposes six procurement tools (get_unread_notifications, get_order_history, search_inventory, create_payable_order, authorize_order, and find_product_alternatives).
  2. Amazon Bedrock (Nova Sonic & Nova Pro): Full duplex bidirectional audio streaming using InvokeModelWithBidirectionalStream with awscrt and AsyncBedrockRuntimeClient, paired with Nova Pro for deep reasoning.
  3. State Machine & Interruption Stack: Integrated into frontend and backend (modality_router.py), maintaining multi-turn context across interruptions and informational detours.
  4. Pure Native Speech-to-Speech (Amazon Nova Sonic): Unlike hybrid systems that cascade speech recognition into an LLM and then into external TTS engines like Amazon Polly, Shopexa streams raw audio bidirectionally into amazon.nova-2-5-sonic. Dropping cascaded TTS eliminated hidden pipeline failures, allowing direct full-duplex speech interaction and native conversational agility.
  5. Echo Show Glassmorphic UI: Built with HTML5/CSS3/JavaScript featuring dynamic waveforms, 3D perspective tilt, and the iconic Alexa glowing ring.

Shopexa Ops Architecture

🚧 Challenges we ran into & Workarounds

  • MCP Streamable HTTP in FastAPI: Mounting the official MCP Python SDK's ASGI app inside FastAPI required coordinating prefix paths and lifespan management. We solved this with dynamic transport detection supporting both streamable_http_app() and sse_app().
  • Acoustic Echo Barge-In Loop: When the smart display spoke through laptop speakers, the browser microphone picked up the voice, falsely triggering barge-in interruptions. We fixed this by gating audio frames during active speech output, adding WebRTC acoustic echo cancellation constraints, and building the Attention Gate classifier.
  • Duplex Stream Teardown Deadlock: When shutting down or reconnecting an InvokeModelWithBidirectionalStream session, smithy_core.aio.eventstream.DuplexEventStream.close() can hang indefinitely waiting on unconsumed frames. We solved this by decoupling stream receipt into a dedicated worker task and applying a strict asynchronous timeout guard on .close().
  • State Loss on Ambient Interruption: Early builds lost in-flight orders when background supplier price spikes fired. We engineered the LIFO Interruption Stack to snapshot active state, render an uncompleted task indicator, and provide voice/touch resumption.

(See our complete friction_log.md for detailed friction entries and official feature requests).

🏆 Accomplishments

  • True Proactive Intelligence: Not just a chatbot that answers questions—an agent that monitors pricing, calculates savings, presents supplier alternatives in a 2x2 grid, and stages purchase orders autonomously.
  • Bulletproof Interruption Resilience: The first smart display agent that can suspend a staged order, answer an inventory side-query, and seamlessly resume where you left off with a spoken voice bridge.
  • Sub-Second Latency: Nova Pro tool executions and Nova Sonic native audio streaming average under 800ms.
  • 100% Automated Test Coverage: 26 / 26 regression tests passing (test_price_alternatives_regression.py) plus dedicated state machine verification (test_resume_state.py).

📚 What we learned

Building for Alexa+ revealed that the future of ambient computing lies in proactive agency combined with attention awareness: an AI that speaks only when it has high-confidence value, stays silent during human-to-human discussion, and preserves user task state through every interruption.

🚀 What's next for Shopexa Ops: Proactive Ambient

Our next step is scaling the backend on AWS. We plan to migrate the local SQLite mock database to Amazon DynamoDB for global scalability, and integrate Amazon EventBridge to listen for real-world supplier webhooks (like Shopify or QuickBooks updates) to trigger our ambient alerts natively in the cloud.

Built With

  • agentic-ai
  • alexa-plus
  • amazon-bedrock
  • amazon-nova-pro
  • amazon-nova-sonic
  • ambient-computing
  • aws-iam
  • awscrt
  • fastapi
  • mcp
  • model-context-protocol
  • python
  • smithy-core
  • sqlite
  • sse
  • streamable-http
  • websockets
Share this project:

Updates

Submission history