Inspiration
Standing in a pharmacy aisle, you face 40 nearly identical boxes, and the one that’s fine for you can be the wrong one for your 72-year-old mom with high blood pressure. Ibuprofen raises blood pressure. Benadryl is on the list of drugs older adults should avoid. Tylenol PM sits right next to Tylenol in almost the same box. Most people pick based on the brand they recognize.
BDBI’s Nomi scores one barcode at a time. We wanted something you just point at the shelf, the medicine cabinet, or a bag of pill bottles, which instantly shows what’s safe for the person who’ll actually take it. That’s Nomifier: nomify everything in view, live.
What it does
Nomifier is an iPhone app (an installable PWA) that puts a green, orange or red AR outline on every box and bottle in view, scored for the person it’s for. The outlines stick to products as you sweep a shelf or pick one up.
Profiles that stay on the phone. Set up “Me” and “Mom, 72, high blood pressure”. One tap recolors everything instantly, with no network call. Tap any product and the frame freezes on a card with a verdict (“Avoid”), a big score, and one plain sentence per risk with its DailyMed source. It also offers read aloud and questions to ask your pharmacist. Safer pick in AR. An animated arrow runs from a red product to a better one in view with the same use. An “I need…” filter (pain, allergy, sleep, cough) dims everything else and stars the best match. Look-alikes handled safely. Tylenol and Tylenol PM stay gray and dashed until you turn the box and its barcode locks the exact product. It never guesses a color. Honest unknowns. A product we don’t recognize is outlined in thin gray. One with an ingredient we haven’t reviewed says “Not rated yet” and is never green. Medicine Cabinet. Add what you already take, and outlines flag duplicates: “Also acetaminophen: you already take Tylenol.” There’s also Compare, scan history, large text, and a sound and pulse when a red product enters the center of the frame. A Grok Bot crew for what no API can do: Aisle Finder finds the safer product, or a cheaper store brand with identical ingredients and strengths, in stock nearby. It holds a pickup order and waits for your Approve or Reject. Coverage Clerk reads an unknown product’s photo, finds it on DailyMed, and opens a schema-validated pull request to add it. Label Watch checks every product’s label and FDA recalls weekly, posts a Grok Page, and puts a ⚠ badge on changed products in the app.
The server only sees camera frames, and the bots only see a product ID, a product photo and a ZIP code. Who the medicine is for never leaves the phone.
How we built it Three laptops, one frozen contract. We split the work by machine: a GPU laptop runs recognition and hosts everything, a “crew” laptop runs Cursor and Grok Bot, and a third builds the app. We froze the WebSocket and REST contract at 7:10 pm, and a mock server let the app get built before the vision pipeline existed. Live recognition (FastAPI on the GPU laptop): YOLOE detects products from text prompts like “medicine box” and “pill bottle”. It’s open-vocabulary, so it needed no training. BoT-SORT tracks each product across frames, compensating for a moving phone. DINOv2 identifies each new track once, against a few catalog photos per product. Products are identified once and then tracked almost for free, which is what makes it real time. The phone streams 640px frames with only one in flight at a time, so frames are dropped rather than queued and lag never builds up. iPhone PWA: React, TypeScript and Vite. It has a native-feeling tab bar, bottom sheets, safe areas and dark mode. It works offline for everything except live recognition, and it’s served over Tailscale HTTPS, since the camera requires a real certificate. Scoring on the phone: we use Nomi’s scoring function and data formats (term tables, personas, conditions, cited hard stops). A TypeScript engine on the phone and a Python twin must agree on the same golden test cases. LLMs never produce a score. Product data: a resolver goes barcode → NDC → openFDA → DailyMed label (active ingredients, strengths, UNIIs) and validates the result with Nomi’s schema package. Every scoring row carries a label section and an exact quote. Grok Bot: webhook and schedule routines, a taught “Aisle Finder lookup” task, computer use, an “Ask first” rule on checkout, GitHub access, and Pages. They talk to a FastAPI and SQLite Bot API with token-checked callbacks. Cursor cloud agents built well-specified backend tasks in parallel, each as its own branch and pull request. Challenges we ran into Look-alike boxes. Tylenol vs. Tylenol PM is a real safety difference that a vision model can easily get wrong. We made ambiguity a feature: a gray “family” outline until the barcode confirms the product. “Safe” vs. “never reviewed.” An ingredient with no risk rows could be either, so we track a reviewed-ingredients list. Unreviewed products are never colored. Barcodes don’t map cleanly to drugs. One UPC can be three possible NDC codes, so we ask openFDA which one exists. Salts differ from their active drug (naproxen sodium vs. naproxen), so we key everything on the active ingredient. iPhone limits. Safari has no barcode API (we used a WebAssembly reader) and no vibration (we used a sound and a visual pulse). The camera also needs HTTPS and a tap to start. Cloud bots, private network. Grok Bots run in xAI’s cloud and can’t reach a private tailnet. We exposed one URL with Tailscale Funnel and bearer-token callbacks. Privacy, enforced. A test fails the build if any request, bot payload or the app’s network module ever contains a persona, condition or medication. Real-time on a moving phone. Running the embedding only on new tracks, and dropping frames instead of queuing them, kept it smooth. Not making things up. Every number had to trace to a label quote or a clinical criterion, which made the knowledge data the slowest part to write. Accomplishments that we’re proud of Live AR on real products, hand-held, with outlines that follow a bottle as the judge picks it up. An instant profile switch: going from “adult” to “72, high blood pressure” recolors the whole shelf in the same frame, with zero network requests. It refuses to guess. Look-alikes and unreviewed products are never given a color they haven’t earned. Health data never leaves the phone, and a test proves it. Bots with brakes: they cite everything, never score, and stop for a human before buying, merging or pushing. Work Nomi can use: a scoring engine, a barcode-to-product resolver and a cited term-table seed, all in Nomi’s own formats, ready to offer to Nomi for its Nov 20 demo day. What we learned Open-vocabulary detection plus DINOv2 matching gets you live product recognition with no training data. The trick is to identify once and track after that. With a frozen contract and a mock server, three people built in parallel all night without blocking each other. Health software is mostly an honesty problem: telling unknown apart from safe, refusing to guess, and citing every number. Agents work best with written specs and hard limits. Cursor took the well-defined code, and Grok Bots took the web tasks no API covers, each stopping for approval. What’s next for Nomifier Pharmacist-reviewed scores, replacing our demo values with Nomi’s reviewed rubric. Grow the catalog through Coverage Clerk pull requests, each one approved by a human. Recognition on the phone itself, so no GPU laptop is needed in the aisle. More people: children, pregnancy and breastfeeding, and more conditions and medications as they’re reviewed. Pharmacist shelf audit: anonymous counts by tier and anticholinergic burden for a whole shelf, published as a Grok Page. Give it back to Nomi: contribute the scoring engine, resolver and data to Nomi as pull requests, with the leads’ OK.
Built With
- grokbot
