Inspiration
I’ve always watched movies or TV shows and wondered, "Where can I buy that jacket?" or "What headphones are those?" Right now, answering that question requires pausing the show, picking up a phone, and searching endlessly across the web.
I built StreamCart to transform the Amazon Fire TV experience from passive entertainment into a frictionless, contextual storefront. By integrating real-time scene analysis with Alexa voice commands, StreamCart bridges the gap between streaming media and e-commerce—allowing viewers to discover, save, or purchase items on screen without ever breaking their immersion or reaching for another device.
What it does
StreamCart turns the living room TV into an interactive shopping experience in real time:
Scene-Aware Overlays: As a movie or show plays, StreamCart tracks scene metadata and dynamically displays non-intrusive product cards whenever a shoppable item appears on screen.
Voice-First Navigation: Viewers can interact hands-free using voice prompts like "Alexa, add to cart" or "Alexa, send to phone".
Cross-Device Handoff: For items requiring size or color selection, StreamCart instantly pushes a deep link to the user's mobile Amazon app.
One-Click Checkout: Leverages existing Amazon payment profiles to enable 1-click purchases directly from the Fire TV interface.
How we built it
Computer Vision & Catalog Pipeline (demo.py): I simulated an automated backend computer vision engine. The script processes individual video frames to extract object bounding boxes and labels, mapping detected items directly to the Amazon Product Catalog (ASINs).
Challenges we ran into
Unobtrusive UX Design: Designing a shopping interface on a TV display is tricky. Traditional pop-ups ruin the viewing experience. I spent significant effort tuning the timing and layout of non-intrusive side drawers so they feel like an organic extension of the stream rather than an ad.
Contextual Scene Matching: Video streams produce rapid frame transitions. Mapping unstructured computer vision output to exact product catalog items in real time requires tight synchronization between timestamp markers and database queries.
Cross-Device State Syncing: Ensuring that an action taken on the TV screen (like saving an item) instantly reflects on the user’s mobile Amazon app without introducing latency required an event-driven data flow.
Accomplishments that we're proud of
Seamless 10-Foot UI: Successfully built an interface that feels natural on a big screen without cluttering the primary video playback.End-to-End Functional Prototype: Delivered a working frontend and simulated computer vision pipeline using pure Python, proving the complete flow from scene detection to cart addition.Voice-Driven E-Commerce: Designed a friction-free shopping flow using voice commands, cutting the average purchase time from minutes of manual searching down to seconds (t<5s).
What we learned
Living Room Interface Paradigms: Designing for the "10-foot experience" requires completely different visual hierarchy and interaction models compared to web or mobile app design.
Metadata-Driven Streaming: I gained deeper insights into how structured scene metadata can add value to media playback far beyond simple subtitles or chapters.
End-to-End Prototyping: I learned how to build and present a full-stack proof of concept rapidly—coupling a functional browser-based frontend with an algorithmic backend pipeline to effectively demonstrate a complex ecosystem vision.
What's next for StreamCart
Live AWS Rekognition Integration: Transitioning from simulated computer vision to live object detection powered by AWS Rekognition Video and Amazon Personalize.
Native Fire OS / Leanback App: Porting the Streamlit prototype to a native Fire OS Kotlin application utilizing the Amazon Fire TV SDK and Video Skill API.
Real-Time AR Try-On: Integrating a mobile companion camera feature where users can scan a QR code on screen to virtually try on clothing or view 3D furniture in their own space using AR before buying.
Log in or sign up for Devpost to join the conversation.