Yorigo: From Cooking Videos to Shoppable Recipes via Nova

Yorigo uses Amazon Nova 2 Lite on Bedrock to transform cooking videos into structured, shoppable recipes — extracting ingredients, steps, and nutrition from ASR and OCR signals in real time.

Comment

Inspiration

Cooking content has exploded on short-form platforms — but watching a video and actually cooking from it are two completely different experiences. We kept seeing the same problem: a creator flies through a recipe in 60 seconds, and viewers are left pausing, rewinding, and guessing at quantities. We built YoriGo to close that gap.

What it does

YoriGo takes any cooking video URL from YouTube, Instagram, or TikTok and automatically produces a fully structured, shoppable recipe. The pipeline validates whether the video is actually about cooking, transcribes audio via ASR, extracts on-screen text via OCR, and sends all signals to Amazon Nova 2 Lite on Bedrock — which structures a clean, metric-based recipe with ingredients, steps, nutrition rating, and taste tags. Structured ingredients are then matched to real products on Coupang and Naver Shopping, enabling one-tap "add to cart" flows.

How we built it

  • AI Core: Amazon Nova 2 Lite via Amazon Bedrock — used for content filtering, recipe structuring, and ingredient price estimation.
  • Backend: FastAPI on EC2, Dockerized, with IAM Role-based Bedrock access.
  • ASR: faster-whisper for fast, accurate audio transcription.
  • OCR: EasyOCR on sampled video frames to capture on-screen ingredient text.
  • Frontend: Flutter mobile app with real-time streaming progress states.
  • Commerce: Coupang and Naver Shopping product mapping for ingredient purchasing.

Challenges we ran into

The hardest problem was prompt engineering for recipe structuring. Cooking videos are wildly inconsistent — creators use colloquial units ("한 줌", "적당히"), skip quantities entirely, or mix languages mid-sentence. We had to encode an extensive set of conversion rules, ingredient normalization logic, and category taxonomies directly into Nova's system prompt to get production-quality output reliably. Balancing latency and accuracy was another challenge — we restructured the pipeline to front-load Nova's content classification step, filtering out non-cooking content before any heavy processing begins.

Accomplishments that we're proud of

  • End-to-end pipeline from raw video URL to structured shoppable recipe, running in production.
  • Nova 2 Lite reliably filling missing ingredient quantities with realistic estimates — a genuinely hard reasoning task.
  • Live commerce integration: users can add all recipe ingredients to their Coupang cart in one tap.

What we learned

Nova 2 Lite's instruction-following on complex, nested JSON schemas is remarkably robust — especially for domain-specific tasks where the output structure is non-trivial. We also learned that combining ASR and OCR signals consistently outperforms single-modality approaches for recipe extraction, particularly for videos where key quantities appear as on-screen text rather than being spoken aloud.

What's next for YoriGo

  • Nova multimodal embeddings for semantic similar-recipe search across our growing recipe database.
  • Nova Act agents to automate shopping cart population across grocery web UIs without requiring platform API access.
  • Expansion beyond Korean grocery platforms to regional e-commerce partners across Southeast Asia.

Built With

Share this project:

Updates