NannyCam

AI-powered remote eldercare that watches medications, restocks the pantry, and guards finances — all through iMessage.

Comment

NannyCam

Inspiration

Watching a family member struggle with early-stage Alzheimer's made us realize how broken remote caregiving is. Adult children call three times a day asking "did you take your pills?" — and still worry. 53 million Americans are unpaid caregivers, and the tools available are either too clinical or require the patient to actively use an app (which defeats the purpose for cognitively-declined individuals). I wanted something that watches passively, acts autonomously, and communicates through a channel every parent already knows — iMessage.

What it does

NannyCam is an AI-powered remote eldercare system built around three pillars:

  • Medication Monitor — A webcam watches the pill organizer. Gemini 2.0 Flash Vision verifies whether the correct compartment was emptied at the scheduled dose time. The caretaker gets an iMessage: "Mom took her 8am Aricept ✓"
  • Autonomous Pantry Restocking — A second webcam scans the pantry. When items are low or empty, NannyCam auto-orders from Walmart via Knot API's AgenticShopping — no human intervention needed for small orders.
  • Bill Protector — Every transaction on the parent's linked card is monitored via Knot TransactionLink. Suspicious charges (over spending limits or in blocked categories) are flagged, and the caretaker can reply "block 42" via iMessage to reject them instantly.

The caretaker never opens an app. Everything flows through iMessage — alerts, approvals, and even a "status" command that returns a daily summary.

How I built it

  • Next.js 16 with App Router for the web dashboard and 12 API routes
  • Gemini 2.0 Flash for vision analysis (medication verification + pantry stock assessment) with Zod schema validation
  • Knot API for TransactionLink (real-time transaction monitoring) and AgenticShopping (autonomous Walmart checkout)
  • Photon Spectrum for bidirectional iMessage — the caretaker texts commands, NannyCam responds with typing indicators and structured replies
  • Supabase for real-time Postgres (live event feed via subscriptions), storage (webcam snapshots), and the outbox table
  • Outbox pattern — Next.js routes never call Photon directly. Messages are enqueued to a DB table and drained by a separate worker process, ensuring crash recovery and no message loss
  • Motion detection — client-side frame diffing (2% threshold for pills, 8% for pantry) with 60-second cooldown to avoid duplicate captures
  • Schedule-triggered captures — pill cam auto-fires within ±10 minutes of each dose time

~5,000 lines of TypeScript across API routes, services, components, and agent worker.

Challenges I ran into

  • Knot sandbox limitations — TransactionLink only returns new transactions since last sync, and the sandbox merchant data is minimal. I built a demo injection endpoint (/api/demo/txn) to simulate realistic flagging scenarios.
  • iMessage cold-start problem — Apple's anti-spam rules prevent messaging users who haven't initiated a conversation. I had to figure out the Photon onboarding flow (email invite → user accepts → bot sends welcome message → thread opens).
  • Two-process architecture — Spectrum's async iterator (for await...of app.messages) can't run inside serverless Next.js route handlers. I split into a Next.js app + a long-lived Bun worker, connected via the outbox table.
  • Gemini vision consistency — pill organizer analysis required careful prompt engineering to distinguish "compartment opened but pills still there" from "compartment emptied correctly."

Accomplishments that I'm proud of

  • End-to-end autonomy — from webcam capture to Gemini analysis to Walmart checkout to iMessage notification, with zero human intervention for routine operations
  • Real-time dashboard — Supabase subscriptions make events appear instantly as they happen, with color-coded status badges that update live when a caretaker approves/blocks from iMessage
  • The "block 42" moment — texting two words on your phone and watching the dashboard update in real-time is genuinely satisfying
  • No app install required — the caretaker's entire interface is iMessage, something every parent and child already has

What I learned

  • The outbox pattern is essential for reliable messaging in serverless architectures — never call external APIs inline
  • Gemini 2.0 Flash is surprisingly good at structured image analysis when you give it a Zod schema to fill
  • Knot's AgenticShopping API can genuinely place orders autonomously, which opens up powerful "agent-on-behalf-of" use cases
  • Building for cognitively-declined users means the patient never touches the tech — the entire UX burden shifts to passive sensors and the caretaker's existing tools

What's next for NannyCam

  • Multi-patient support — one caretaker managing multiple family members
  • Fall detection — adding a room camera with pose estimation to detect falls and alert emergency contacts
  • Medication refill tracking — using Knot TransactionLink to detect pharmacy purchases and cross-reference against prescription schedules
  • Voice integration — allowing the patient to say "I took my medicine" as a secondary confirmation signal
  • Production deployment — moving from Knot sandbox to live TransactionLink with real bank connections

Built With

  • bun
  • gemini-2.0-flash
  • knot-api
  • lucide
  • next.js-16
  • photon-spectrum
  • react
  • react-19
  • supabase
  • tailwind-css-4
  • typescript
  • zod
Share this project:

Updates