Inspiration
For millions of people with chronic conditions, dining out is a minefield. A patient on Warfarin can suffer a health crisis just by eating a spinach salad, while someone on MAOIs risks a reaction from aged cheese. Existing apps are passive—they just log pills. We built Foodie to be an active biomedical bodyguard.
What it does
Foodie connects to a user's Dedalus identity to securely load their medication schedule. When a user uploads a restaurant menu, Foodie scans it to flag specific interactions (e.g., 'Spinach blocks Warfarin') and filters the menu to show only safe options.
How we built it
But we didn't stop at advice. We implemented Real Computer Use. Once a user selects a safe dish, our SafetyKit Agent (powered by Playwright) launches a live browser instance. It navigates to the restaurant's reservation portal, inputs the guest's details, and types a precise, medically accurate note to the kitchen (e.g., 'Guest on Warfarin. Pre-ordering Grilled Salmon. Strict isolation from Vitamin K rich foods required'). This automates the anxiety of 'the allergy talk' and ensures the kitchen gets the right data, every time.
Challenges we ran into
Our biggest hurdle was the "Computer Use" implementation with SafetyKit. Getting an autonomous agent to reliably navigate third-party reservation portals was difficult because DOM elements change, and pop-ups (like cookie banners) often broke the flow. We had to write robust error-handling logic in Playwright to ensure the agent could recover if a button wasn't immediately clickable.
Another significant challenge was Menu Parsing accuracy. Restaurant menus are unstructured, messy, and often use fancy fonts. Early tests with standard OCR often missed ingredients listed in small print. We had to refine our image processing pipeline to specifically look for "danger keywords" (like Grapefruit, Spinach, Aged Cheese) and map them back to the user's Dedalus medical profile without generating false positives.
Accomplishments that we're proud of
Building a True Agent, Not Just a Chatbot: We are incredibly proud of the SafetyKit Agent. Seeing the browser physically open, type the reservation, and inject the medical safety note without human hands touching the keyboard was a "magic moment" for the team.
The "Green Light" System: We moved beyond just saying "No." We're proud of the logic that suggests safe alternatives (e.g., "Avoid the Ravioli, but the Grilled Salmon is safe"), which empowers patients rather than restricting them.
What we learned
We learned the power of "Human-in-the-Loop" AI. While our agent automates the booking, we realized the importance of showing the user the specific note ("Strict isolation from Vitamin K") before sending it. Trust is built when the user sees the AI advocating for them correctly.
What's next for Foodie
Wearable Integration: We plan to sync real-time biometrics (like heart rate or glucose levels) to offer dynamic food recommendations based on the user's current physical state.
Voice-Activated SafetyKit: We want to upgrade the agent to handle phone reservations using voice synthesis for restaurants that don't have web portals.
Expansion to Grocery: Applying the same scanning technology to barcode scanning in supermarkets, instantly flagging ingredients on the back of packages that conflict with the user's Dedalus profile.
Log in or sign up for Devpost to join the conversation.