TokShop Lens

Turn vertical videos shoppable instantly. Multimodal AI automatically overlays verified product tags, tracks creator affiliate commissions, and hosts an interactive assistant sales rep.

Comment

Inspiration

Video shopping is massive, but finding the exact products featured in vertical videos is highly frustrating. Viewers are forced to exit the app to search external e-commerce sites, leading to search drop-offs and missed conversions. We built TokShop Lens to make vertical videos completely interactive and instantly shoppable.

What it does

TokShop Lens takes any uploaded review or GRWM video, runs a Google Gemini 1.5 Flash multimodal analysis background job to identify products and timestamps, and grounds them in a matching catalog:

  1. Interactive Overlays: Product cards appear directly over the video in sync with timestamps.
  2. AI Sales Rep: An in-player chat companion answers specific product questions using video context.
  3. Affiliate Shopping Bag: Tracks checkout cart totals and logs a 15% affiliate commission directly to the creator.

How we built it

  • Frontend: Next.js 16 (built with Turbopack), Tailwind CSS, Lucide icons, responsive layout.
  • Backend: FastAPI framework, Python, SQLite database.
  • AI Processing: Google Gemini 1.5 Flash multimodal API.
  • Matching Algorithm: Hybrid matching combining character-based SequenceMatcher and word-level Jaccard token overlap with category boosts.

Challenges we ran into

  • Server Timeouts: Heavy video analysis took longer than standard HTTP request windows, so we built an asynchronous background polling task executor.
  • Fuzzy Product Matches: Catalog items are written differently than AI captions. We engineered a token-character hybrid matching algorithm to boost category alignment scores.

Accomplishments that we're proud of

  • Building a fluid, mobile-native vertical video player interface.
  • Integrating a copyable creator affiliate referral tracker.
  • Ensuring the system builds with zero errors and has backend rate-limiting protection.

What we learned

  • Designing and implementing asynchronous API polling loops.
  • Optimizing SQLite database transactions to prevent concurrent locks.
  • Working with Google Gemini's Multimodal File API constraints.

What's next for TokShop Lens

  • Live Stream Parsing: Mapped overlay tagging on real-time TikTok streams.
  • Affiliate Smart Payouts: Automatic smart contract commission payments to creators.
  • Dynamic Inventory Checkout APIs: Direct, secure multi-vendor cart integration.

Built With

Share this project:

Updates

Submission history