Inspiration
Finding keys, a phone or a familiar mug can be a small task with a big impact on someone's independence. When an object is difficult to see or distinguish, a simple search can become frustrating or require help from another person.
Lookout explores how an iPhone can make that everyday interaction easier. Our intended audience includes people with blindness, low vision, colour vision differences, and temporary or situational difficulties such as misplaced glasses. These needs vary, so we are designing around the task of finding an object rather than assuming one experience of vision.
Our mission is simple: less seeking, more living.
What it does
Lookout lets you ask for an everyday object by voice and uses the live camera view to look for it. Spoken guidance describes its direction, while a haptic beacon helps you follow the search without relying on the screen. On supported Pro iPhones, LiDAR supplies distance information.
You can also teach Lookout a personal object, such as your wallet, so future searches can use its saved visual representation. The app supports spoken feedback, VoiceOver announcements and management of taught objects.
Privacy is central to the design. A camera used around a home can see personal belongings, documents and family members. Lookout processes its object-finding pipeline on the phone, with no cloud inference. Speech assets need to be installed during setup before offline use.
How we built it
We built a native iPhone prototype with Swift and Apple's on-device frameworks. Apple Vision proposes candidate object crops. MobileCLIP running through Core ML compares those crops with the spoken request or saved representations of taught objects.
A smaller model searches for candidates, while a larger model checks a promising match. Foundation Models parses requests on supported devices, with a rule-based fallback. Camera input, speech, haptics and LiDAR depth work together to turn a match into guidance a person can follow.
The pipeline prioritises recent frames so processing does not build up a queue of outdated camera views.
Challenges we ran into
The main challenge was turning recognition into useful guidance: deciding when a match is confident enough to announce, avoiding repeated speech, coordinating VoiceOver and haptics, and responding when an object is lost from view.
Similar-looking belongings are particularly difficult. Teaching a wallet does not automatically guarantee that every lookalike will be rejected. We also had to account for differences between devices, limited processing and memory budgets, and the distinction between relative depth estimates and metric distance.
Accomplishments and lessons
We brought voice queries, visual matching, personal-object teaching and multimodal guidance together in a working iPhone prototype. We are proud of making privacy and nonvisual interaction part of the core design.
The project reinforced that recognising an object is only one part of accessibility. The timing, wording and consistency of feedback matter just as much. A model result becomes useful only when the person can understand it and act on it.
What's next for Lookout
Our next priority is co-design and testing with people across the vision needs we aim to support. We want to measure everyday task completion, recognition reliability, feedback clarity and performance across devices and environments.
We also plan to improve lookalike discrimination, refine teaching and recovery flows, and review model licensing before broader distribution. Lookout is an experimental object-finding aid; this prototype does not establish effectiveness for every vision condition or replace mobility and safety tools.
Log in or sign up for Devpost to join the conversation.