Aura Health

Aura health is an AI-powered progressive web app (PWA) designed to help users make safer and more informed food choices by analyzing their grocery or restaurant receipts in real time.

Comment

🌌 Mission — Aura Health:

Aura Health started from love—the kind that shows up in everyday moments. We’re the ones reminding our parents to take their meds, checking ingredients before we serve a meal, quietly googling to keep everyone safe. As first-gen, we’ve often been the translators and researchers, trying to make the “right” health choices without clear guidance. Aura is our way of saying: you don’t have to do this alone. Whether you’re scanning a receipt, getting a gentle heads-up, or asking Astrea for a smarter swap, we want everyday decisions to feel lighter, kinder, and rooted in care—not fear. It’s a small act of love you can carry in your pocket.

Inspiration:

Grocery receipts are the most granular, real-world record of what we actually buy—yet we toss them. We wanted a zero-new-habit way to turn that data into guardrails: quick health cautions, dietary nudges, and answers grounded in what’s literally in your cart. Snap → extract → analyze → act.

What it does:

  • Aura Health is a PWA that:
  • Lets you upload or snap a receipt and runs OCR to extract items.
  • Computes a health score and flags (e.g., sugar/sodium cautions, basic drug–food risks).
  • Offers Astrea, a context-aware chat grounded in your latest receipt.
  • Optionally sends a weekly Health Digest email with trends and suggestions.
  • It’s fast to try (installable PWA), safe by design (server-side keys, CORS, validation), and model-agnostic (OpenRouter, Gemini, Claude ready).

How we built it:

  • Frontend (React + TypeScript)
  • Vite + shadcn/ui + Tailwind for a clean, mobile-first dashboard.
  • Tabs: Upload/Camera, Receipt Analysis, Chat with Astrea.
  • PWA install + offline shell; responsive layout and accessible components.

Why React (not Next.js) for a hackathon?

  • Fewer moving parts (no SSR/route handlers) → faster iteration and simpler deploys.
  • Static asset hosting works everywhere; easier PWA setup and local dev.
  • Straightforward Vite dev server and proxy for /api/*.

Backend (Python + FastAPI):

  • /api/ocr/process accepts multipart/form-data (image_data) → Tesseract → normalized items.
  • /api/ai/chat accepts { messages, model, receipts } → model router (OpenRouter / Gemini / Claude) → grounded responses.
  • /api/health liveness; Swagger at /docs for quick tests.
  • .env for OPENROUTER_API_KEY, GEMINI_API_KEY, CORS_ORIGINS, TESSERACT_PATH.

Newsletter (Retention):

  • Job composes markdown→HTML digest (trends, warnings) and sends via Resend.

Gemini Live (real-time guidance):

  • What it is: A streaming mode for on-the-spot answers while you scan or review a receipt.
  • How it runs: FastAPI session + SSE streaming to the UI; grounded with the current receipt JSON; uses Gemini API (primary) with graceful fallback to standard chat.

Astrea — receipt-grounded AI chatbot: Purpose: Clear, actionable answers about what was purchased and how to improve it. Models: Routes via OpenRouter and Gemini API (model-agnostic prompts; provider fallback).

What it does:

  • Explains flags (e.g., “Why the grapefruit warning?”) with sources.
  • Targets improvements (“Cut ~15g added sugar without raising total cost”).
  • Summarizes receipts into a weekly Health Digest (trends + suggestions).

How it stays reliable:

  • Structured grounding (parsed items, categories, nutrition estimates).
  • Rules engine for allergens/contraindications; model for mapping/explanations.
  • Scoped history to the active receipt or digest window.

Example:

  • While scanning a grocery receipt, a user asks: “Find a lower-sodium swap for ‘Campbell’s Chicken Noodle’, under $3, same aisle.”
  • Gemini Live returns two alternatives with sodium per serving and price deltas, lets the user tap Add swap to save it to their list—without leaving the scan flow.

Challenges we ran into — Team Stories:

Alisha (frontend & stack choice): “We started with Next.js + TS but realized the SSR and file-based routing added complexity we didn’t need for a PWA. Switching to React + Vite made hot-reloading and API proxying painless—and let me focus on UX instead of framework plumbing.”

Jasleen (email + deploys): “Integrating Resend and actually receiving emails was brand new to me. The first sends failed silently, but once I got the DNS + domain verified and moved env keys correctly, seeing that first Health Digest in my inbox was so rewarding. I also deployed the backend—Vercel config + custom domain wiring was stressful, but finally connecting everything was ridiculously satisfying.”

Aysha (OCR + analysis): “Our scanner kept misreading line items, which wrecked the health analysis. I pivoted to feed raw text to the Gemini API and let the model infer product matches. After a bunch of merging strategies and prompt tweaks, our flags got way more accurate.”

Marilyn (Gemini Live + Astrea chat): “I fought an accidental while-loop bug and couldn’t get chat history to persist with the right receipt context. Fixing the state flow and grounding messages to the active receipt finally made conversations feel coherent.”

What actually bit us (and fixes):

  • CORS & ports: Frontend on 5173/5174 blocked by backend → explicit CORS_ORIGINS + preflight headers in FastAPI.
  • Env & keys: 401s from model providers → mounted .env, /health check, and “keys loaded” log on boot.
  • Tesseract path: Worked locally, failed in container → install OS package in image + TESSERACT_PATH=/usr/bin/tesseract.
  • Payload size/timeouts: Large images → client-side compression + server limits; “Processed Sample” fallback for live demo.
  • Proxy routing: /api/* in dev/prod → Vite proxy in dev; path-based routing in deploy (Railway/Render/Vercel).
  • Accomplishments we’re proud of
  • Shipped our first AI-powered web app end-to-end: installable PWA + FastAPI backend + multi-model chat grounded in live OCR.
  • A clean API surface judges can exercise in Swagger in seconds.
  • Model router that hot-swaps between OpenRouter/Gemini for reliability under hackathon rate limits.
  • A habit loop via the weekly digest, so value shows up even when users forget to open the app.

What we learned:

  • Grounding > vibes: Structured receipt items in prompts reduce hallucinations and keep advice concrete.
  • “Instant” ≈ <2s: Predictable latency (small images + early feedback UI) beats raw throughput.
  • OCR is 80/20: Tesseract is great for clean prints; deskew/denoise + normalization unlock the last mile.
  • Infra can tank a demo: /docs and a preprocessed sample are lifesavers.
  • Accessibility compounds: Keyboard flows and clear warnings improve UX for everyone.

What’s next for Aura Health:

  • E-receipts & loyalty: ingest digital receipts (email forwarding or retailer APIs) to skip photos and boost accuracy.
  • Product ontology & nutrition DB: brand-level mapping for better scoring and fewer ambiguities.
  • On-device pre-processing: compress/deskew pre-upload for speed and cost.
  • Stronger safety signals: curated drug–food sources + confidence scoring.
  • Personalization: goals/preferences that tune scores and recommendations.
  • Privacy & control: one-tap “delete my data,” clearer scopes, optional fully local mode.
  • Timely nudges: push notifications and in-app prompts at the cart.

Built With

Share this project:

Updates