AIKiosQ with Amazon Nova

Meet Sparky: Your AI master electrician. Powered by Amazon Nova, AIKiosQ identifies parts via camera, provides expert repair advice, and locates inventory in real-time. DIY expertise, on-demand.

Comment

🛠️ Project Summary & Purpose

AIKiosQ is an interactive, voice-first digital assistant designed for brick-and-mortar hardware stores. Featuring "Sparky," a veteran electrician persona with decades of experience, the kiosk bridges the gap between expert staff shortages and customer needs.

  • 🚶 Motion Detection: Customers simply step up to the kiosk (which automatically detects motion to initiate the session) and speak naturally to Sparky.
  • 📸 Computer Vision: If a customer brings an unknown hardware part to the store, they can hold it up to the kiosk's camera.
  • 🧠 Instant Identification: The AI identifies the item, provides step-by-step installation instructions, checks the store's local inventory, and displays the aisle map to help the customer find the product.

Goal: To enhance the retail customer experience through immediate, knowledgeable, and multimodal AI assistance.

🚀 Leveraging Amazon Nova Foundation Models

Our solution is deeply integrated with Amazon Nova foundation models, aligning closely with both Voice AI and Multimodal Understanding focus areas:

🎙️ 1. Amazon Nova Sonic (amazon.nova-2-sonic-v1:0)

Real-Time Voice AI: At the core of the kiosk is Amazon Nova Sonic, driving the real-time speech-to-speech interactions.

  • The Stack: We implemented a Node.js backend utilizing HTTP/2 bidirectional streaming to connect our React kiosk client directly to Bedrock.
  • Performance: By streaming 16kHz PCM audio bidirectionally, we achieve fluid, ultra-low latency conversational interactions.
  • Persona: Sparky's friendly personality and deep electrical domain knowledge are entirely enforced via Nova Sonic's system prompts.

👁️ 2. Amazon Nova Lite 2 (us.amazon.nova-2-lite-v1:0)

Multimodal Vision: When a customer asks Sparky to look at a physical part, Nova Sonic triggers a native tool call.

  • Capture: The kiosk captures a high-resolution image of the item and sends it to Amazon Nova Lite 2 via the Converse API.
  • Analysis: Nova Lite 2 analyzes the complex image, identifies the exact hardware component (e.g., a "Burnt Outlet"), and extracts safe handling instructions.

🔧 3. Complex Agentic Tool Use

AIKiosQ goes beyond basic chat by orchestrating complex physical-to-digital workflows using Nova's native tool calling. Nova Sonic intelligently decides when to invoke:

  • analyze_part (Triggering the camera)
  • check_inventory (Querying the local store database)
  • show_aisle (Mapping the physical store)

💡 Conclusion

By combining Nova Sonic's industry-leading voice capabilities with the multimodal reasoning of Nova Lite 2, AIKiosQ demonstrates a practical, highly-innovative enterprise solution that delivers direct business value to the retail sector.

Inspiration

I needed to replace a three way light switch. I went to a big box hardware store for help and when I finally found someone to help, they didn't know how to wire the switch and told me I should hire a licensed electrician. Also they didn't know if they had the part in stock and directed me to the wrong Aisle.

How I built it

I started with an existing project that was designed for plumbing parts. It was using Google Gemini models and Google Cloud Run. I wanted to show how easy it was to completely change the project to be designed for electrical repairs instead of plumbing repairs. Also I wanted to show how easy it is to switch from Google Gemini to Amazon Nova models and AWS App Runner.

Challenges I ran into

🎙️ AI Greeting Limitation (Nova Sonic)

The Problem: Amazon Nova Sonic requires an initial audio input stream to trigger response generation; a simple text-based "Hello" from the system is not sufficient to start the conversation.

🔌 Socket.IO Connection Stability (The "Reboot" Loop)

The Problem: AWS App Runner's Application Load Balancer (ALB) often fails to reliably forward the WebSocket Upgrade header. This causes the handshake to fail, leading to a relentless re-connection loop that mimics a server crash.

  • The Fix: We reconfigured the Socket.IO client to utilize "Polling" as the primary transport.
  • The Result: This ensures the connection begins with standard HTTP—which the ALB handles natively—before attempting a stable upgrade to WebSockets, eliminating the connection "crash."

Accomplishments that I'm proud of

  • I incorporated automated tests including playwright which tests the program like a human.
  • I Incorporated automated deploy script to send everything to AWS App Runner.
  • Both of these are described in the bottom of the README.md

What I learned

  • I learned how to use Amazon Nova models
  • I learned how to host the application on AWS App Runner

What's next for AIKiosQ with Amazon Nova

  • Implement Acoustic Echo Cancellation (AEC): Add Acoustic Echo Cancellation (AEC) to prevent the microphone picking up the output from the speaker and accidently interrupting itself.
  • Expand Domain Knowledge: This project demonstrates Sparky the electrician. Multiple personas for each part of the hardware store should be created. Utilizing Amazon Nova's capabilities to dynamically switch personas or route queries based on the part shown (e.g., from "Sparky" the electrician to "Mac" a plumbing expert) would maximize the kiosk's enterprise value across the entire store.
  • Secure Enterprise Integrations: For the check_inventory and show_aisle tools, ensure the architecture is designed to securely query live enterprise ERP or inventory management systems. I could enhance the AWS architecture by utilizing Amazon API Gateway and AWS Lambda to securely manage these internal data fetches.

Built With

Share this project:

Updates