homecookd

Preserve family recipes in the voices of the people who made them. Cook hands-free from their real clips, and share the dishes with your community.

Comment

Inspiration

As second-generation immigrant children, one of the biggest divides we face is staying connected to our culture when our families are thousands of miles away. Food is how our families say “I love you,” but that bond gets stretched thin by distance, language barriers, and different upbringings.

Our family recipes were never written down or methodically measured. They live in our parents’ and grandparents’ hands and voices, and every year, more of them disappear.

Most AI pulls people toward screens. We wanted AI that brings families closer. So we built homecookd: to preserve family recipes in the voices of the people who made them, and to share them with entire communities.

What it does

homecookd turns a family member cooking at home into a recipe anyone can cook, keep, and share.

  • Record a recipe: A parent or grandparent cooks like they normally would, in their own language, while wearing our homecookd glasses or with a phone propped up nearby. Our assistant, homie, asks quick questions during the quiet moments, like how much of an ingredient they used or who taught them the dish
  • Turn it into easy steps: homecookd breaks the video into short, simple steps. Each one comes with the clip where it happened, so you can see and hear exactly how they did it
  • Cook hands-free: Anyone can move through the recipe by voice. Say “next,” “repeat,” or ask a question like “when do I add the onions?”, and the app plays the cook’s own answer from the video
  • Made for everyone: Voice controls, big captions, and simple prompts mean older family members who don’t use much tech can record and cook without any help. It also works when your hands are covered in ingredients while cooking
  • Share with your community: Post a recipe for people nearby, find other families’ dishes, and host or join a dinner around one. Guests can save the recipe with credit to the family it came from, and the original cook gets to see how many people have made it
  • Private by default: Every recipe stays within your family until you choose to share it. Voices and videos need their own permission, and we only ever show approximate locations

How we built it

  • Hardware: Our glasses are regular frames with a thumb-sized ESP32-S3 board (camera, microphone, and Wi-Fi) clipped to the arm, powered by a battery pack in the cook’s pocket. They record the cook’s first-person video and voice, time-synced, and a small Google Cloud VM stitches the footage into an MP4
  • Mobile app: Built with Expo and React Native and shipped to iPhones through TestFlight. The cook records and uploads, the person cooking along follows by voice, and both see live updates
  • Uploads and storage: A small upload signer on Google Cloud Run gives the phone a safe one-time link to upload each video to Google Cloud Storage. Raw videos stay in a private bucket; the cut clips and a public recipe file (with no family stories in it) go in a separate media bucket
  • Backend: Convex is our main database. It holds families, people, sessions, recipes, processing jobs, questions, messages, recordings, photos, voice notes, and stories and tips (private to the family), and pushes live updates to both phones, so a new recipe shows up the moment it’s ready. Apple sign-in and invites run on our existing Supabase setup, which we froze; Convex copies the family over the first time a phone connects
  • Video pipeline: A Python and ffmpeg worker claims each new job from Convex, prepares the video, cuts the clips, runs the AI steps, and publishes the recipe, reporting each stage back so the cook can watch progress live. For the hackathon it runs on one of our laptops
  • Understanding the video: Gemini watches the video and notes what the cook said and did, with timestamps, then puts the steps in order and shortens them. It also hears the cook live in their own language (Marathi in our demo), translates between that language and English, and reads yes-or-no answers, and it stays on as a backup writer and backup voice
  • Recipe writing: Muse Spark writes the final recipe, stories, and tips in English while keeping the cook’s own words, and ties each step to its clip. It also drafts the note a kid sends back with a photo of their finished dish and writes the Community story cards
  • Finding the right moment: Moss indexes each family’s recipe moments, one index per family, so a spoken question finds the right clip in milliseconds. It only finds moments and never writes answers
  • Live voice service: A small always-on service on Google Cloud Run runs the live cooking conversation and homie while the cook records. Claude makes the judgment calls: on each turn it decides what the person cooking wants (next step, a timer, a question, or “ask the cook”) and checks whether Moss’s match actually answers it, without putting words in the cook’s mouth. While the cook records, Claude decides whether homie should ask something, and it turns their answers into an English line in their name, like “Grandpa says…”
  • Voice: Cartesia Ink 2 turns speech into text, both for the person cooking and for the cook’s voice-note answers, and Cartesia Sonic gives homie its warm, unhurried voice for reading steps and replies aloud. homie only speaks when the cook doesn’t have a clip for the question
  • Website: A Next.js site on Vercel (homecookd.app) hosts the landing page, recipe share pages, family invites, and dinner invites
  • Community and privacy: Recipes are private until shared, one at a time. Locations are rounded to about 1 km before they’re stored, so we only ever show approximate distance, and a dinner’s address appears only after the host accepts you. API keys stay on the server side, in Convex, Google Secret Manager, and Supabase, never in the app

Challenges we ran into

  • Egocentric data: Fitting an ESP32-S3 onto a pair of glasses so it was comfortable to wear, while keeping both the video and audio feed clean, was a major hurdle. We optimized the firmware and tested different transmission protocols to keep the stream steady and time-synced, since every recipe step depends on matching what the cook said to what they were doing
  • Long cooking videos: A real cooking session can run 30 to 50 minutes, which is too long for a single model call. We split each video into 10-minute ranges with 30-second overlaps, ran Gemini on them in parallel at low resolution, had Gemini merge them into one ordered, shortened list of steps without duplicates, and then had Muse Spark write the final recipe
  • Voice latency: Every spoken question goes through Cartesia Ink 2 (speech-to-text), Claude, Moss, and back out through Cartesia Sonic (text-to-speech), and each step adds delay. We kept the voice service always on in Google Cloud Run and let simple commands like “next” skip Claude entirely, so answers feel instant
  • Homie Hearing Itself: The glasses’ mic picked up homie’s own voice, which would have ended up in the transcript. We filtered homie’s audio out of the live pass so it only listens to the cook
  • Knowing When to Talk: An assistant that interrupts someone mid-chop or mid-story ruins the moment. We gave homie strict timing rules: wait for a lull, back off after an ignored question, and never interrupt a story

Accomplishments that we're proud of

  • Truly building something that makes the people we love smile. homecookd was born out of our shared struggles as immigrant children trying to give back to the communities that made us who we are today, and being able to see the smile on our families' faces as they use the app is one of our proudest moments
  • Wearable hardware in a weekend: We went from a bare ESP32-S3 board to glasses that record time-synced, first-person video and audio, all without soldering.
  • An end-to-end pipeline: Raw first-person footage goes in, and a structured recipe comes out, with every step linked to its exact clip and the cook’s original words kept alongside it
  • Answers in the cook’s real voice: The live voice loop matches a spoken question to the right moment in the video and plays that clip. When Moss doesn’t find a confident match, Claude says so instead of guessing and sends the question to the cook
  • Privacy in the data model: Stories and tips live in Convex and stay private to the family, the public recipe file in storage leaves them out, recipes are private until shared, and locations are rounded to about 1 km before they’re ever saved

What we learned

  • User-Friendly Dev: Identifying homecookd's users as both older individuals and younger students meant making the UI of the app accessible for these demographics through simplified buttons and features such as voice control and translation
  • Human-Centric Dev: Building for a wide variety of cultures and people, especially with intricate data like family recipes, requires making a conscious effort to diversify the types of data being trained and the way it is trained to ensure every demographic is represented
  • One Job Per Model: Giving each AI a narrow task and passing structured JSON between steps made the system easier to debug. It also meant Gemini could step in as a backup if Muse Spark returned something our app couldn’t read

What's next for homecookd

  • Bringing this to campus communities: We want to start with cultural clubs and international student groups, so students far from home can find a taste of it through dinners hosted nearby
  • More languages: We want every family to teach in their own words, so we’re expanding voice support beyond our demo languages and testing with more families who don’t speak English
  • Two-way questions: Right now, recipes go from the cook to the family. Next, homie will carry family members’ questions into the cook’s next recording (“Shreya asked how you get it so crispy”)
  • Lighter, sturdier glasses: We want to shrink the hardware into something families can keep at home and wear comfortably through a full cook, with longer battery life
  • Recipe swaps and family archives: We want families to swap dishes with each other and build a private, searchable archive of every recipe and story, so nothing gets lost when the person who made it is gone

Built With

Share this project:

Updates