Inspiration
Private Eye grew out of Mark's supervised research on interpretability for vision-language models (VLMs) at Worcester Polytechnic Institute and UW-Madison.
We wanted to know whether we could use VLMs to help blind people interact with their surroundings better. Most assistants for blind people either upload each photo to a server or connect the user to a sighted volunteer. We wanted one that answers at once, works with no signal, and keeps everything the camera sees on the phone.
What it does
Private Eye is an iPhone app for blind and low-vision users. You point the phone, tap once, and it tells you what is in front with the nearest things first: "Canon EOS camera with a zoom lens on the desk. Laptop behind it, to the right." Hold the screen and you can ask any question about what the camera sees, have it read text, or ask "How far is that?" and get an accurate distance from LiDAR. We have optimized the responses to include concise information on distance, direction, and properties of the objects ahead to be as useful to visually-impaired users as possible.
It also remembers. Say "This is Sarah" once, and it finds her in a crowd a month later. Say "Remember this as my water bottle", and from then on it points out your bottle among the others on the table, and tells you where and when it was last seen. And it can look for things: ask and it starts a 30-second search that beeps each second, then chimes and buzzes when the desired object comes into view. The search works for almost anything you can describe. In testing it found "a T-shirt that has code on it."
No data leaves the phone. There is no account and no upload, and the memory of faces and belongings is a single encrypted file on the device.
How we built it
The app is written in Swift for the iPhone 17 Pro and uses the camera and LiDAR. The language model is LFM3-VL-2B, a 2-billion-parameter VLM, quantized to 4 bits and run on the phone with llama.cpp. It loads in about a second, reads a frame in about 0.7 seconds and speaks at 40 to 50 tokens a second. Around it sit YOLOE for detection and segmentation, two embedding models for memory (ArcFace for faces, DINOv3 for belongings), and on-device text recognition, speech recognition and voice.
The model on the phone is Witness-2B, our own fine-tune of LFM3, and we built it on Databricks. We started from VizWiz, 20,000 photos taken by blind people, each with the question they asked. gpt-6.1-sol rewrote every label from the photo through the Databricks AI Gateway. The labels, photos, grades and model versions live in Unity Catalog. Training was two LoRA runs on a single GPU, 13 and 25 minutes. One Databricks Job grades any model on 500 held-out photos, with the runs tracked in MLflow and an AI/BI dashboard and Genie on top.
Challenges we ran into
We had a working prototype by the first evening, and in our own testing it did fairly well. Then we ran it on 500 photos from blind users and found the small model hallucinating. Asked "What is my blood pressure reading?" on a display washed out by glare, it answered "125/80". On impossible questions, it invented an answer 58% of the time.
So we analyzed the data, and discovered problems there. Blind users can't check the shot, so 31% of the photos can't answer their question: they are blurry, dark, or something important is out of frame. The crowd-written labels were often inaccurate. Our relabeling overturned 14% of them and refined another 37%.
The first fine-tune was a mixed success. It cut invented answers from 58% to 9%, but it also refused 71 photos it could have answered, and it lost their sense of position since this wasn't tested in the dataset. For the second run we added 3,000 scene descriptions, drawn from COCO, VizWiz and our own phone frames, and 6,400 refusals rewritten to say what to do next.
The last challenge was getting data for improving the features. We reached out to blind people for feedback but only one person responded, so we had to stand in for them. That is not a substitute, but deeply considering what our users would want did lead us to shorten the descriptions and add search.
Accomplishments that we're proud of
We are proudest of Witness-2B. On photos that can't answer the question it invents an answer 8% of the time. The six other 2B models we tested, stock LFM3 included, invent one 57% to 80% of the time, so Witness-2B hallucinates seven times less than the next best. It is also the most accurate of the seven overall, with under half the errors of any rival: 13% of its answers are wrong, against 31% to 39%.
We also believe that some of the most impressive features are the ability to look for items and to notify the blind user when they are detected. Additionally, we believe that the ability to remember objects should be the most useful to visually impaired people.
What we learned
The biggest gain came from fixing the data, with clean labels considerably improving the model. We also learned that fine-tuning can break capabilities just as easily as it improves them: our first run looked better on the benchmark but had obvious problems in the hand, so we don't take a better benchmark score at face value.
We learned that a refusal has to be useful. "Can't tell" helps nobody; "Turn the box over" does. And on a phone you can't buy accuracy with a bigger model. A small model with the right training beat every other model its size.
What's next for Private Eye
Testing with blind users would greatly motivate feature changes. After that we want to move the camera to smart glasses so both hands stay free. We also think using LiDAR to map indoor spaces so the app can find a path through a building could be useful. We already partially utilize the LiDAR sensor, but its usage could definitely be expanded.
