Thumbnail
Perceive, decide, act. Only vision can say “found”.
OpenCV 5 on AWS, from phone to answer.
No look-alike ever called “found”.
0/11 → 9/11 after fine-tuning.
Your bottle, detected and checked against your photos.
High contrast and large text for low vision.

Inspiration

For a blind person, a lost cup or set of keys means sweeping both hands over every surface, or asking someone for help. Apps like Seeing AI or Lookout can say "a cup", but a home has several cups. The real question is: where is my cup? That gap, from a thing to your thing, is what FindMine tries to close.

What it does

FindMine is a phone web app (one link, nothing to install). You register an item once: say its name and hold it, and the app takes 5 photos by itself when the picture is sharp. Later you say "find my keys", and FindMine:

coaches the camera (too dark, blurry, lens covered, too shaky),
detects objects of that kind,
checks it is yours by comparing with your 5 photos, and asks "is it yours?" when it is not sure,
guides your hand with clock-face speech ("2 o'clock, arm's length"), stereo beeps and vibration.

It speaks English, French and Arabic, works with screen readers, and has large-text and dark modes.

How we built it

All vision at inference time runs in OpenCV 5.0.0 on AWS EC2:

Coach: Laplacian variance, histograms, optical flow, Sobel.
Detect: YOLO11n fine-tuned on 12 everyday classes (COCO + LVIS, then a keys fine-tune), exported to ONNX and run in cv.dnn, with a 2×2 zoomed-tile search for small items.
Is it mine? DINOv2 embeddings in cv.dnn + SIFT matching with RANSAC + HSV colour histograms, fused with a logistic model trained with cv.ml on our own photos.
Track: TrackerCSRT + KalmanFilter.
Guide: clock direction and distance band.

The search is an agent: a state machine decides on every frame, and Claude Haiku 4.5 on Amazon Bedrock is called only when the search is stuck. It can speak, turn on the torch or ask the user, but it can never say "found"; only the vision scores can. Every decision is traced and viewable at /trace/<session>.

AWS: EC2 (c7i-flex.large), S3 (face-blurred registration photos, models), DynamoDB (items), Bedrock (planner), Route 53 health checks + CloudWatch alarm and dashboard. Training ran on Kaggle GPUs, never on AWS.

Results

0 of 75 look-alike items (same kind, someone else's) were ever called "found" on our own home photos.
Keys: the first detector recognised our keys in 0 of 11 close-ups; after fine-tuning, 9 of 11.
Live round trip from Algeria to us-east-1: 300–400 ms per frame.
A real tester search: "is it yours?" → yes → guided by beeps → found in 20 s.

Challenges

Yours vs a look-alike from far away is hard. Up close our verifier separates them well; from across the room it can't. So instead of guessing, FindMine says "move closer so I can check it's yours". Safety first: below the found threshold it always asks.
Small objects. Keys are a few dozen pixels at search distance, so we added a zoomed-tile search.
A fine-tune that failed. We trained a third detector for wallets, earbuds and medicine on public datasets. It looked great on their test images (mAP50 0.32 → 0.41), but on our photos it only recognised wallets up close and invented wallets in 45 of 322 frames. We kept the previous model. Public product photos teach the close-up look, not the small object in a room that a search actually sees.
iPhone speech only works if it starts inside a tap, so we reworked how the app unlocks its voice.

What we learned

Measure on your own data before shipping a model, report failures honestly, and let the user confirm when the system is unsure. For people who can't see the screen, a confident wrong answer is worse than a question.

What's next

Training frames that look like real searches (small objects in rooms), a real blindfolded A/B study, Darija voice commands, and reading medicine labels aloud.