Inspiration
Sight Companion started through my long-term collaboration with Light Into Europe, an organisation supporting visually and hearing-impaired people in the UK and Romania.
Rather than starting with a technology idea, I began by asking visually impaired people a simple question:
What tools do you actually use, and where do they still fall short?
At the time, several people I spoke with were using tools such as Be My Eyes, where another person could help interpret what the camera or an image was showing.
That service was useful, but the conversations made me wonder:
How much of that assistance could technology provide immediately, privately and independently?
I started with a simple capability: allowing the user to take an image and receive an AI-generated description of what was visible.
But once visually impaired users began trying Sight Companion, the project evolved.
Their feedback helped me understand that describing an image was only the beginning.
The more interesting questions became:
- Can some useful capabilities work without an internet connection?
- Can the phone identify what matters without sending every image to the cloud?
- Can it tell the user how many people are nearby without identifying them?
- Can it help recognise products while shopping?
- Can it help the user locate something rather than simply describe a photograph?
- Can the system react continuously as the environment changes?
Today, Sight Companion is being used and tested by a small group of approximately 3–4 visually impaired users, and I continue developing the application based on their needs and feedback.
That user-driven evolution is what led to Sight Companion Live.
What it does
Sight Companion Live is a privacy-conscious visual copilot for blind and visually impaired people.
Instead of only answering:
"What is in this picture?"
the project is evolving toward answering:
"What does the user need to know or do right now?"
The core competition experience is Find & Guide.
A user might ask:
"Find the door."
"Where is an empty chair?"
"Help me find the checkout."
"What is directly ahead of me?"
Sight Companion analyses the camera feed and determines what information is useful for the current task.
If the requested object is not visible, it may ask the user to move the camera:
"Slowly scan to your right."
It then observes the new camera frames.
When relevant visual evidence appears, the guidance changes:
"Stop. The door is ahead and slightly to the right."
This creates a continuous loop:
User goal → Perception → Decision → Guidance → New perception → Updated decision
Rather than describing everything visible, Sight Companion Live aims to communicate what is most relevant to the user's immediate objective.
Existing capabilities and evolution
Sight Companion began with AI-powered image description.
From there, I started investigating capabilities that could increase independence while reducing dependency on continuous internet connectivity.
One area was shopping assistance.
I added barcode scanning so a visually impaired user could identify a product using the phone camera. This opened a broader question: could Sight Companion eventually help the user actively locate a product rather than requiring them to already be holding it?
Another area I began exploring was offline visual processing.
For example, anonymous people detection could allow the application to communicate:
"Three people are in front of you."
or:
"The area appears crowded."
without needing facial recognition and, where possible, without sending the camera feed to a remote service.
The OpenCV AI Competition provides an opportunity to take these experiments much further and turn Sight Companion from an image-description application into an active visual assistant.
How we are building it
For this competition, Sight Companion Live is being developed around a hybrid edge-and-cloud architecture.
OpenCV 5 and local perception
OpenCV 5 will perform substantive processing of live camera frames, including areas such as:
- frame preprocessing and enhancement
- region-of-interest extraction
- object and person localisation
- tracking
- motion analysis
- barcode localisation
- geometric and perspective correction
- camera-frame quality assessment
- preparation of visual evidence for downstream AI reasoning
Where practical, immediate and privacy-sensitive perception will happen locally on the device.
This is especially important because an assistive application should remain useful even when network connectivity is poor or unavailable.
Agentic Vision
Computer-vision results are not intended to simply produce another description.
They become observations that affect what the system does next.
For example:
Goal: Find a chair
↓
No chair detected
↓
Action: "Slowly move the camera to the right."
↓
New camera frames
↓
Chair detected
↓
Action: "Stop. A chair is visible ahead."
This perception-to-action loop is central to Sight Companion Live.
Depending on what the camera observes, the system can:
- continue searching
- ask the user to reposition the camera
- provide a directional cue
- communicate a relevant nearby event
- request another view when confidence is insufficient
- invoke deeper AI reasoning when necessary
AWS and privacy
AWS will provide meaningful cloud capabilities for higher-level reasoning, orchestration and supporting application services.
However, Sight Companion Live is deliberately being designed so that continuous video does not need to be uploaded unnecessarily.
The architecture follows a simple principle:
Edge for speed, privacy and offline capability. Cloud for deeper intelligence.
When cloud processing is useful, the system can send selected frames, cropped regions or derived visual information instead of continuously streaming everything the camera sees.
This is particularly important for an application that may be used in homes, shops and other everyday environments containing sensitive visual information.
User-centred development
The most important part of Sight Companion has been developing it with feedback from the people it is intended to support.
A small group of visually impaired users currently use and test the application.
Their feedback influences what gets built next.
That process has changed my own view of the product considerably.
At first, the technical question was:
"How well can AI describe an image?"
Through working with users, it became:
"What information actually helps someone complete an everyday task more independently?"
Those are very different product questions.
The second one is now driving Sight Companion Live.
Challenges
One major challenge is information overload.
A camera can detect many things simultaneously, but a visually impaired user does not necessarily benefit from hearing every object that the system sees.
The application therefore needs to understand context and prioritise information.
Another challenge is uncertainty.
Poor lighting, motion, occlusion and unusual viewpoints can all affect computer vision. In an assistive application, confidently communicating incorrect information may be worse than saying that the system is uncertain.
Sight Companion Live therefore aims to use confidence-aware behaviour and ask the user for another view when visual evidence is insufficient.
A third challenge is privacy.
Continuous camera usage can reveal highly personal information. That is one reason I am particularly interested in expanding Sight Companion's offline and on-device perception capabilities.
Responsible use
Sight Companion Live is intended as supplementary environmental assistance.
It is not intended to replace a white cane, guide dog, orientation and mobility training, or other primary mobility aids.
The competition implementation will avoid facial recognition and will focus on anonymous environmental awareness.
The system will also be designed to communicate uncertainty rather than presenting every computer-vision result as fact.
What we are evaluating
For the OpenCV AI Competition 2026, I want to evaluate Sight Companion Live through repeatable real-world tasks rather than showing only carefully selected successful demonstrations.
Planned evaluation areas include:
- finding selected objects such as doors and chairs
- time required to locate a target
- visual-search success rate
- people counting accuracy
- barcode localisation and recognition
- local processing latency
- cloud response latency
- offline capability
- false detections
- failure and recovery behaviour
Feedback from visually impaired users will remain an important part of the evaluation and product-development process.
What's next
The immediate goal for the competition is to build a reliable Find & Guide experience where live visual perception genuinely changes what the assistant does next.
Beyond the competition, the same approach could support richer shopping assistance, indoor contextual awareness, public-space accessibility, haptic feedback and wearable interfaces.
Sight Companion began as a way to describe an image.
Working directly with visually impaired users has made the ambition much bigger:
Help people obtain the right visual information, at the right moment, with greater privacy and independence.
Built With
- accessibility
- agentic
- agents
- ai
- amazon-web-services
- api
- artificial
- computer
- flutter
- intelligence
- multimodal
- opencv
- python
- rag
- rest
- search
- speech
- vector
- vision
Log in or sign up for Devpost to join the conversation.