Freshly

Inspiration

Every household throws away food it forgot it owned. The pattern is always the same: you buy vegetables with a vague plan, the plan never materialises, and two weeks later you find them at the back of the fridge. Meanwhile you order takeout, because deciding what to cook is a harder problem than cooking itself.

We noticed that the tools meant to help are all split across that gap. Recipe apps suggest dishes without knowing what you have. Pantry trackers tell you what you have without suggesting anything. Meal planners assume you'll go shopping for everything from scratch. Nobody closes the loop.

Freshly was built to close it:

Photo → pantry inventory → AI-generated recipe → weekly meal plan → shopping list

Each stage reads the one before it. The recipe generator knows your actual pantry contents and your weight goal. The shopping list subtracts your pantry from your plan, so it only ever lists the gap — never the onion already sitting in your kitchen.

What it does

You add ingredients by photographing them, scanning a barcode, or typing them in. The photo path is the interesting one: the image goes to a multimodal model that returns the ingredient name, its food category, a storage tip, a confidence score, and — when it's legible on the packaging — the expiry date. Your pantry then shows expiry badges (Expired / Today / 4d left), so the food that needs cooking first is the food you see first.

From there you generate recipes constrained by what you own, filtered by meal type, cuisine, servings, and free-text notes like "something spicy, under 20 minutes." Each recipe comes back with ordered instructions, per-serving macros, the full ingredient list, and a separate missingIngredients array. You drop dishes into a 7-day × 3-meal planner — or let the backend propose an entire de-duplicated week and re-roll individual meals you don't like — and the shopping list rebuilds itself from the plan.

How we built it

Mobile app. React Native 0.85 on Expo SDK 56, Expo Router v6 for file-based routing, TypeScript in strict mode, Reanimated 4 and Expo Haptics for motion and feedback, with the React Compiler enabled.

Backend. NestJS 11 over PostgreSQL 16 via TypeORM, organised by domain module — auth, pantry-items, recipes, meal-plans, meal-recommendations, shopping-list, and shared ai and mail modules. Passport JWT with access and refresh tokens, refresh tokens stored hashed rather than in plaintext. A JwtAuthGuard is registered globally as an APP_GUARD, so every route is authenticated unless explicitly marked @Public() — the safe default is locked, and opening a route is a deliberate act. A global ValidationPipe runs with whitelist and forbidNonWhitelisted, so unknown request fields are rejected rather than quietly ignored. Eleven versioned migrations, synchronize off.

The AI layer is the part we're most pleased with architecturally. Both LLM calls sit behind interfaces — IAiVisionProvider and IRecipeGenerationProvider — resolved through a config-driven factory keyed on an AI_PROVIDER environment variable. No service ever imports a vendor SDK. Swapping a model or an entire provider is one new file and one line in the factory, with zero call-site changes. That decision paid for itself the first time we changed our minds about which model should handle which job.

Recipe generation uses Gemini 2.5 Flash via the Gemini API, called with a responseSchema that pins the exact output shape — the full ingredient list, the missingIngredients subset, ordered instructions, and per-serving nutrition. The prompt injects the pantry contents, the user's height and weight, and their goal translated into dietary guidance (gain → higher-calorie and protein-rich; lose → calorie-controlled and high-satiety). Vision runs on a separate multimodal model at temperature = 0.1, because reading an expiry date off a label is extraction, not creativity — we want the same answer every time.

The shopping list is a set operation rather than a list of suggestions. For a planned week $P$ of recipes and a pantry $I$, each line item is:

$$\text{needed}(x) = \max\left(0,\ \sum_{r \in P} q_r(x) \;-\; q_I(x)\right)$$

Lines are merged by (name, unit), rows the plan no longer needs are dropped, and items you've already ticked stay ticked across rebuilds — so editing your plan on Wednesday doesn't wipe the shopping you did on Monday.

Challenges we ran into

Getting structured data out of a language model. Our first recipe implementation asked for JSON in the prompt and hoped. We got markdown code fences, occasional prose preambles, string quantities where we needed numbers, and missingIngredients that cheerfully included things sitting in the pantry. We fixed it in layers: pin the schema at the API level rather than in the prose, spell out the units enum explicitly, and state the missing-ingredient rule as a constraint rather than a description. Moving to schema-enforced structured output let us delete our markdown-stripping code entirely — a whole class of parse failures stopped existing.

Multipart uploads through an interceptor stack. Our API client attaches a JWT and a JSON content-type to every request, which silently corrupted every photo upload — setting Content-Type manually destroys the multipart boundary that the client generates. Uploads failed with an unhelpful 400 and no clue why. The fix was small; finding it was not. The client now strips the JSON content-type on multipart requests specifically.

Staying logged in without staying logged in forever. Short-lived access tokens meant users got bounced to the login screen mid-task. The interceptor now catches a 401, silently refreshes the token, and replays the original request — the user never sees it. Combined with exponential backoff on network and 5xx failures (3 attempts, delay capped at 8 seconds, i.e. $\min(2^n, 8)$) and a cached pantry for offline reads, the app survives a bad connection instead of collapsing on it.

Connecting a physical phone to a laptop backend. The classic hackathon time-sink: hardcode your machine's LAN IP, then rewrite it every time you change networks. Instead we read Expo's manifest hostUri, strip the dev-server port, and point at port 3000 on the same host. Any device on the same Wi-Fi connects with no configuration at all.

Merging duplicate pantry items. Buying milk twice shouldn't create two rows. But if the two cartons have different expiry dates, silently picking one is a food-safety decision the app has no business making quietly. Multi-select merge now prompts explicitly to resolve conflicting expiry dates rather than guessing.

Temp files and failure paths. Uploaded photos land on disk via Multer, get read to base64, go to the model, and are deleted in a finally block — regardless of whether the model call succeeded, failed, or timed out. A missing multipart part returns a clean 400 instead of crashing the handler.

What we learned

The biggest lesson was that prompt engineering is schema engineering. Every hour we spent rewording instructions bought less reliability than a single well-specified response schema. Constrain the output shape at the API level, and the model's remaining freedom is exactly the creative part you actually wanted from it.

The second lesson was about dependency boundaries under time pressure. Writing two interfaces and a factory felt like ceremony on day one of a hackathon. It stopped feeling like ceremony the moment we swapped a provider and touched exactly two files. Abstraction earns its keep precisely when you're moving fast, because moving fast means changing your mind.

And a smaller, more practical one: make the safe thing the default. A globally registered auth guard means a forgotten decorator leaves a route locked, not open. A validation pipe with forbidNonWhitelisted means an unexpected field is a loud 400, not a silent write. Both caught real mistakes during the build.

What's next

Three areas are scaffolded — entities, controllers, and CRUD routes exist — but have no logic behind them yet: push notifications for expiring food, the weekly nutrition log that would close the loop between the meal plan and the user's goal, and YouTube recipe import for pulling a dish out of a video you've just watched.

Beyond that: getting off the developer machine. The backend already has a production multi-stage Dockerfile, so containerised deployment is a short step. The mobile app still needs an EAS build profile and store configuration.