Lodestone

An iMessage agent that turns your receipts into recall checks: it saves what you bought and verifies the exact affected model against official CPSC notices.

Comment

Inspiration

Recall notices are public, but they do not answer the question a household actually has: is the thing in my closet affected?

A notice lists model numbers, date codes, and affected ranges. A receipt says something like MYCHARGE PWR HUB 49.99. Closing that gap means finding the right notice, locating tiny codes on the product, comparing them by hand, and remembering to follow up. It is easy to postpone or forget.

We wanted that workflow to happen where people already communicate: their messages. Text a receipt, confirm the product, and check it against an official source.

What it does

Lodestone is an iMessage purchase assistant. Every purchase, on record.

  1. Text a receipt photo. Gemini extracts visible product details and leaves unreadable fields empty instead of guessing. Multi-item receipts support numbered selection or ALL.
  2. Confirm what it read. Products enter the saved inventory only after you reply YES. You can correct extracted details with commands such as EDIT MODEL ….
  3. Opt in with MONITOR. Lodestone records your preference and performs an initial CPSC recall search, returning candidate notices with official links.
  4. Verify the identifiers. When a possible notice requires more information, send a label photo or type something like MODEL AO10FK-A DATE 1521.
  5. Act on the result. Lodestone shows the evidence, official source, and stated remedy. Commands such as DONE, RETURNED, and REMOVED record your follow-up.

Completed checks distinguish affected, possible match requiring verification, and no match against the notice or sources checked. A no-match result is never presented as proof that a product is safe. Source failures are reported as incomplete checks.

Our exact-verification demo uses the historical myCharge POWER HUB recall, CPSC recall 24-271, clearly labeled as historical demonstration data. Other CPSC candidates remain possible matches requiring review.

Lodestone also tracks user-provided return, warranty, and price-adjustment dates, lists upcoming or overdue actions, and generates copyable merchant support drafts. It does not verify merchant policies or submit claims.

A companion evidence desk website displays saved purchases, capture sources, recall evidence, recovery dates, and follow-up status.

How we built it

  • Messaging: A TypeScript gateway using the spectrum-ts SDK listens to the iMessage stream, handles text and photo messages, enforces a sender allowlist, and forwards normalized events to the backend.
  • Backend: Python and FastAPI manage the conversation state machine, inventory, repeated-event deduplication, and follow-up actions.
  • Extraction: Gemini 2.5 Flash returns structured JSON. Its instructions treat image text as data and prohibit guessing unreadable identifiers.
  • Recall retrieval: The CPSC SaferProducts Recall REST API supplies official candidates.
  • Verification: Deterministic rules determine the historical demo result. Name similarity retrieves candidates; it does not establish that a product is affected.
  • Storage: Supabase PostgreSQL stores operational records, with an in-memory mode for local demos.
  • Website: React, TypeScript, Vite, and custom CSS power the evidence desk.
  • Optional analytics: An asynchronous Snowflake copy supports purchase insights, with Cortex summaries when configured.

For our historical demo notice:

$$ \text{affected} \iff m \in M \land d \in D $$

Here, (M) contains AO10FK-A, AO10FK-B, and AO10FK-C. The date code (d) uses four-digit WWYY format. (D) includes valid week codes ending in 21 or 22, plus 0123, 0923, 1523, and 1723.

Notices without implemented verification rules remain possible matches with official links.

Challenges we ran into

  • The data was not where we expected. For our demo notice, the structured model field was empty, while affected models appeared in the description. We learned to compare against the notice text.
  • “Date” did not mean purchase date. Users naturally entered a calendar date, but the recall requires a four-digit manufacturing code. We added specific guidance explaining where to find it.
  • O versus 0. An extraction error such as reading AO10FK-A as A010FK-A can change the result. Identifier confirmation became an essential part of the workflow.
  • A stalled request blocked later messages. The gateway processed messages sequentially. We added bounded backend request timeouts and user-facing retry messages.
  • Conversation state needed flexibility. Follow-up prompts initially swallowed commands such as RETURNED and LABEL. Unrecognized text could also lose the selected product context. We corrected these paths and added regression tests.
  • External services introduced practical limits. During testing, we encountered Gemini free-tier quota limits and local networking exhaustion that disrupted outbound connections.

What we learned

AI is useful for reading messy photos, while explicit rules make identifier decisions understandable and testable. A fuzzy retrieval score is not the probability that a product is recalled.

We also learned that uncertainty needs a useful next step: “Possible match—here is the identifier I still need.”

Finally, messaging agents need reliable state, deduplication, and timeouts. Users do not always follow the expected script, and the conversation must preserve their progress.

What’s next

  • Scheduled rechecks and proactive alerts for opted-in products.
  • More CPSC notices with tested identifier-verification rules.
  • FDA and USDA food recall coverage.
  • Authenticated personal evidence desks.
  • Reliable analytics retries, backfills, and reconciliation.
  • Merchant-policy sources and richer recovery workflows built on the date tracking and support drafts already implemented.

Built With

Share this project:

Updates