Mise

Mise extracts social recipes via local AI, wrapped in a dark acid-lime UI with a morphing Command Pill, syntax highlighting, reactive scaling & wake-locked Focus Mode. 100% offline & private.

Comment

Inspiration

It started with utter kitchen chaos. Over the years, our family recipes became scattered across a messy labyrinth of digital and physical noise: dozens of bookmarked TikTok videos, saved Instagram Reels, bookmarked YouTube Shorts, random web links saved in browser tabs, camera photos of physical heirloom cookbooks, and half-typed ingredients in Notes with no structure, no searchability, and no sense to it. When we tried using existing recipe management apps, we were met with another frustrating wall: compulsory cloud account sign-ups, aggressive subscription paywalls, intrusive targeted ads, and hidden tracking scripts that sell health, meal intake, and personal dietary habits to third-party ad networks. I built Mise (from the French culinary term "Mise en place" - everything in its place) to solve both problems at once. I wanted a high-performance extraction engine that effortlessly converts chaotic multi-source culinary inputs into structured, interactive recipes, while remaining 100% offline, zero-cloud dependent, and privacy-first.

What it does

Mise is a privacy-first cooking companion for iOS and Android built around an elevated "Carbon & Acid Forest" dark aesthetic (#121415 Anthracite background with #DFFF00 Electric Lime accents) and a thumb-friendly Contextual Morphing Command Pill bottom dock. The core of Mise is its On-Device Local LLM Engine (litertlm). Instead of relaying sensitive user data or raw text to expensive cloud LLM endpoints, Mise runs quantized local Large Language Models directly on your device's NPU/GPU to parse, extract, structure, and normalize raw recipe data.

1. On-Device Local LLM Parsing & Extraction Engine

  • Local LLM Heavy Lifting (litertlm): The central brain of Mise is a quantized local LLM running directly on-device via LiteRT. It takes chaotic audio transcripts, unstructured camera OCR text, or scraped web content, and parses them into standardized, clean recipe schemas.
  • On-Device JSON Repair Pipeline: Local LLMs can sometimes emit minor formatting flaws. Mise implements an automated client-side JSON repair pipeline to guarantee zero-crash execution and deterministic recipe parsing.
  • Speech-to-Text via Local model: Video audio (TikTok, Instagram Reels, YouTube Shorts etc) is transcribed locally using an embedded GGML C++ model resampled at \( \text{Sample Rate} = 16000\text{ Hz} \) before being fed into the local LLM.
  • Camera OCR Scanner: Point your phone camera at a physical cookbook to extract raw text via Google MLKit, which the local LLM immediately transforms into structured Markdown. ### 2. Interactive Recipe Canvas & "Culinary Syntax"
  • Culinary Syntax Highlighting: Treats recipes like source code. Using a custom markdown renderer, ingredient measurements are highlighted in translucent Electric Lime pills, cook times in Soft Cyan, and cooking temperatures in Digital Peach for instant 1-second visual scanning under bright kitchen lighting.
  • Reactive Serving Scaler: Tap the serving multiplier to scale ingredient quantities in real-time across the entire recipe without layout jitter: $$ Q_{\text{target}} = Q_{\text{base}} \times \left( \frac{S_{\text{target}}}{S_{\text{base}}} \right) $$
  • Instant Unit Conversion: Switch seamlessly between Metric (\( g, mL, ^\circ\text{C} \)) and Imperial (\( oz, \text{cups}, ^\circ\text{F} \)) systems using dynamic unit transformation matrices. ### 3. Nutrition Profile Engine & Macro Math
  • Real-Time Macro Math: Calculates total energy and macro splits (Protein, Carbs, Fat) rendered in Apple-Fitness-style vector macro rings.
  • Dynamic Macro Scaling: Macro values automatically update alongside serving size adjustments: $$ \text{Macro}{\text{total}} = \sum{i=1}^{N} \left( m_i \cdot \frac{S_{\text{target}}}{S_{\text{base}}} \right) $$ where \( m_i \) represents the baseline macro count for ingredient \( i \). ### 4. Hands-Free Cooking "Focus Mode"
  • Wake-Locked Step Cards: Tap the "Play" icon to enter landscape Focus Mode. Screen timeout is disabled so your phone never goes dark mid-recipe.
  • Ergonomic Tap Navigation: Large single-step cards let you advance steps by tapping anywhere on the right half of the screen, perfect for messy or flour-covered hands.
  • Tactile Haptics: Light haptic impact feedback confirms step completion so you don't need to stare at the screen while cooking.
  • Spatial Pull-to-Close: Drag down on the recipe canvas to drop the card back into your Cookbook grid. ### 5. Smart Meals-Based Shopping Lists & Taxonomy Tagging
  • Multi-Recipe Aggregation: Select multiple recipes for the week (e.g., Spicy Garlic Noodles + Chicken Parm), and Mise automatically consolidates duplicate ingredients (e.g., \( \frac{1}{2}\text{ cup} + \frac{3}{4}\text{ cup} = \frac{5}{4}\text{ cups} \) of olive oil).
  • Smart Taxonomy Tagging: Organize your cookbook with custom tags like #high-protein, #meal-prep, #keto, or #under-30-mins. ### 6. Smart Pantry Engine
  • Aisle Categorization: Automatically categorizes shopping items into produce, dairy, meat, and pantry sections for fast grocery store navigation.
  • Pantry Matching: Compares your current pantry stock against saved recipes to show what percentage of ingredients you already possess. ### 7. Zero-Cloud P2P Sync & Encrypted Local Storage
  • Local Network P2P Sharing: Share recipes instantly with family over local Wi-Fi using mDNS service discovery and an embedded HTTP server, completely serverless.
  • Encrypted Local Database: All user data, meal logs, and recipes are stored in an ObjectBox binary store protected with AES payload encryption.

How I built it

  • Core Flutter & Dart Architecture
  • 1. On-Device Local LLM Parsing Engine (litertlm)
  • 2. Local GGML C++ Audio Inference (whisper_ggml_plus)
  • 3. Riverpod 3.0 Reactive State Engine
  • 4. ObjectBox AES-256 Encrypted Store (objectbox)
  • 5. Serverless P2P Discovery (bonsoir + shelf)
  • Framework: Flutter (Dart) targeting iOS and Android natively.
  • Local LLM Engine: litertlm and tflite_flutter executing quantized local LLM inference directly on mobile hardware.
  • State Management: Riverpod 3.0 with reactive annotations for state propagation across serving sizes, unit toggles, and nutrition calculations.
  • Audio AI & Speech Recognition: whisper_ggml_plus executing C++ GGML bindings for local Whisper audio inference.
  • Computer Vision & OCR: google_mlkit_text_recognition for instant offline camera extraction.
  • Data Layer: Encrypted ObjectBox local database with Dart FFI bindings.
  • Network & Sharing: bonsoir for zero-configuration mDNS discovery and shelf / shelf_router for local P2P HTTP streaming.
  • Screen & Device Controls: wakelock_plus for display management and sensors_plus for motion handling.

Challenges I ran into

  • On-Device LLM Context & Memory Optimization: Running local LLM inference (litertlm) alongside Whisper speech recognition on mobile RAM limits required strict memory management, model quantization, and custom JSON repair logic to avoid out-of-memory crashes on midrange phones.
  • Jitter-Free Reactive Markdown: Standard Markdown packages re-render text blocks completely when variables change, causing ugly layout shifts. I developed a custom MarkdownElementBuilder that intercepts ingredient quantity tags and wraps them in AnimatedDefaultTextStyle widgets, allowing numbers to cross-fade seamlessly when scaling serving sizes.
  • Fractional Unit Aggregation Math: Combining ingredients from multiple recipes required building a fraction simplification parser to add non-standard culinary units
  • Serverless P2P Handshakes: Establishing zero-cloud P2P sync across iOS and Android required handling strict local network permission prompts (Bonjour on iOS, NSD on Android) and resolving IP collisions gracefully.

Accomplishments that I'm proud of

  • Zero Cloud Server Costs (\( \$0.00 \) COGS): I successfully executed 100% of application AI processing, including local LLM recipe parsing, audio transcription, camera OCR, and database encryption, directly on-device. This achieves a net gross margin of 85% without spending a single cent on cloud API servers or third-party GPU infrastructure.
  • 100% Deterministic Local LLM Parsing: Engineering an automated client-side JSON repair pipeline that intercepts and corrects minor formatting anomalies from local LLM outputs, achieving a perfect parse success rate
  • Production-Grade Kitchen Ergonomics: Creating the Carbon & Acid Forest high-contrast dark design system (#121415) and the morphing Contextual Command Pill bottom dock, allowing home cooks to navigate recipes effortlessly with messy or flour-covered hands.
  • True Serverless P2P Recipe Sharing: Implementing peer-to-peer recipe streaming over local Wi-Fi networks using mDNS and an embedded HTTP server with zero reliance on cloud signal relays or central databases.

What I learned

  • Zero-Cloud COGS Economics: By executing speech recognition, OCR, local LLM parsing, and storage on-device, I proved that complex consumer AI apps can operate with \( \$0.00 \) in cloud server API overhead per user, yielding over \( 85\% \) gross margins
  • On-Device AI is Production-Ready: Quantized local LLMs are more than capable of handling domain-specific extraction tasks locally with zero latency or cloud network reliance.
  • Kitchen-First Ergonomics: Design for the physical environment. Large tap targets, dark backgrounds (#121415), and wake-locked single-step cards dramatically improve usability when hands are dirty or wet. ## What's next for Mise
  • WearOS & watchOS Kitchen Companion: Syncing step timers directly to your smartwatch so you can step away from the stove.
  • Receipt Scanner Ingestion: Auto-populating the Smart Pantry by snapping a photo of your paper grocery receipts via local LLM parser.

Built With

Share this project:

Updates

Submission history