CookAlong

Your Fire TV becomes a hands-free kitchen: scan your fridge, let an AI agent plan dinner, then cook step by step with timers and your voice.

Comment

Inspiration

Every evening starts the same way: open the fridge, stare at it, then scroll recipes on a phone. Cooking from a phone means greasy fingers, tiny text, a screen that keeps locking and a timer app you keep losing. Yet most homes already have a big, bright screen you can read from across the kitchen: the TV. Recipe apps are built for phones, so we built one for the TV, and made it start from what's actually in your fridge.

What it does

CookAlong is a native Fire TV app for the whole cooking evening, driven by the remote or your voice.

  1. Fridge scan (computer vision). The TV shows a QR code. Scan it with your phone, then snap your fridge or pick a photo from your gallery. Amazon Nova Pro on Amazon Bedrock lists the ingredients in about two seconds, and the TV ranks all 585 recipes by what you can cook right now: "Ready to cook" and "Almost there", with exactly what's missing. It understands Indian and international names such as capsicum, curd, brinjal and atta.
  2. Plan dinner (an AI agent). One press hands the ingredients to an agent built with the Strands Agents SDK on Amazon Nova. It searches the recipes with real tools, suggests swaps for missing items and writes a shopping list. The TV shows one to three meals, the list and a tip, and reads the plan aloud. A Bedrock Guardrail keeps it on cooking.
  3. Cooking mode (multi-modal). One big step at a time, readable from across the room, with up to four timers, a chime, step animations and resume after a restart. Every step is read aloud in a natural Amazon Polly voice; offline, the TV's own voice takes over. The shopping list goes to your phone with one scan.
  4. Voice. On Fire OS 8, "Alexa, next / previous / pause / play" drive the steps and timers through the app's media session. The Cook Along Alexa skill goes further: after a one-time six-digit code it handles "go to step 4", "set a timer for five minutes", "how much paneer do I need" (scaled to your servings), recipe search, "what's for dinner" and "ask the chef".

The library has 585 recipes, 178 of them Indian, from openly licensed sources (based.cooking, Wikibooks Cookbook, HowToCook), with sources and photo credits listed in the repository.

How we built it

The TV app is Kotlin with Jetpack Compose for TV and needs no Google services, so it runs on Fire OS 6 and newer. Home is a streaming-style layout: a full-bleed billboard that follows your focus, and poster rows you browse with the D-pad. Recipes sync from DynamoDB and are cached on the TV, so saved recipes and cooking mode work offline.

The backend runs on AWS in Mumbai (ap-south-1), and the app holds no API keys:

Phone photo → Lambda (fridge scan) → Amazon Nova Pro on Bedrock → ingredients
Plan dinner → Lambda (Strands agent on Nova) → tools read DynamoDB → plan checked by the server
Read aloud  → Lambda /tts → Amazon Polly (generative voice) → cached on the TV per sentence
Alexa       → Alexa skill → Lambda → DynamoDB command queue → TV polls once a second
  • cookalong-fridge-scan serves the recipe catalog, runs the fridge scan, and hosts the TV bridge and the /tts endpoint for Polly.
  • cookalong-agent runs the dinner planner with four tools: find_recipes, get_recipe, suggest_swap and shopping_list. After the agent answers, the server re-checks every fact the TV will show: the recipe exists, diet and time limits hold, and the minutes and missing items are right. The model chooses and explains, but it can't invent a recipe.
  • Amazon Bedrock Guardrails keep the agent on cooking, block medical and allergy-safety advice and prompt attacks, and mask emails and phone numbers.
  • cookalong-alexa handles the skill. Fire TV apps can't receive pushes, so the skill queues commands in DynamoDB and the TV picks them up while CookAlong is on screen.
  • Amazon DynamoDB (three tables with TTL) holds recipes, scans, pairing codes and per-TV limits. AWS CloudFormation (backend/template.yaml) describes the stack, and each Lambda has its own least-privilege IAM role.

Costs stay small by design: a scan needs a live pairing code, each TV gets 12 plans an hour, and the whole service has a daily cap. By Nova list prices a scan is under 1 cent and a plan about 1 to 3 cents (estimates). The code has 82 app unit tests and 78 backend tests.

Challenges we ran into

  • No Claude on an India-billed account. The Bedrock Marketplace subscription failed, so we moved everything to Amazon Nova and measured Nova Pro against Nova 2 Lite on real food photos.
  • A silent Fire TV. The stick's default text-to-speech engine had no voices, so read-aloud was silent until the app tried every installed engine. Amazon Polly then gave it a natural voice, with the TV's voice kept as the offline fallback.
  • Linking Alexa to a TV app. There's no API that links an Alexa account to a Fire TV app, so we built a six-digit pairing code and a polling queue. Real devices taught us two more things: Alexa hears "pair" as Bluetooth, so the phrase became "my code is …", and a skill in development only answers its owner's account, so other people need a beta invite.
  • Trusting an AI planner. A model can name a recipe that doesn't exist or break a diet, so every plan is validated against the database before the TV shows it.
  • Shopping lists that make sense. Early lists told people to buy water, salt and optional garnishes. Staples and optional items are now recognised and never added.
  • TV details. Rows slid under the top fade while scrolling, the screensaver started in the middle of a recipe, and long steps clipped beside the timers. Testing on a real stick found each one, and each is fixed.

Accomplishments that we're proud of

  • A complete pipeline from a phone photo to an agent-planned dinner to hands-free cooking, running on real Fire TV Sticks.
  • An agent that uses real tools and is checked by the server, not a single prompt.
  • A 10-foot interface that feels like a streaming app, with 585 credited open-licence recipes.
  • A natural Polly voice online that falls back to the TV's own voice offline, so cooking never stops.

What we learned

  • Amazon Nova Pro calls tools reliably and reads food photos well in about two seconds.
  • Guardrails are a cheap way to keep a cooking assistant on topic.
  • On a TV, focus and scrolling details decide whether an app feels good, and only a real device shows them.
  • Fire OS versions differ: "Alexa, next" reaches an app's media session on Fire OS 8 but not on Fire OS 7.

What's next

  • Kitchens without a TV: keep the cooking session in the skill and add an Alexa Presentation Language screen, so an Echo Show can show the step, ingredients and timers by itself.
  • Publish the Alexa skill, replace polling with Amazon Device Messaging, add more regional recipes, and port to Vega OS.

Honest status

  • Tested on two real Fire TV Sticks (Fire OS 8 and Fire OS 7) and an Android TV emulator: install, Home, fridge scan, plan dinner, cooking with timers and Polly read-aloud. The demo video was filmed on a real stick.
  • "Alexa, next" and "Alexa, previous" work by voice on Fire OS 8 through Fire TV's media controls, with no skill involved.
  • The Cook Along skill is not published. It works by voice on the developer's own account and was tested end to end through Alexa's simulator, moving the real TV. To try it on your own Echo or Fire TV, email [email protected] for a beta invite.
  • The CloudFormation template validates, but the live resources were created with the deploy scripts. The agent runs on Lambda; Amazon Bedrock AgentCore is not used.

Built With

Share this project:

Updates

Submission history