Inspiration

NOVA was created from a simple idea: blind people should be able to understand and interact with their surroundings more independently.

We wanted to build more than a system that simply detects objects. The goal was to create an AI Vision assistant that can understand what is around the user, describe important visual information, remember where things are, and communicate the results through voice.

This idea became NOVA: a smart vision assistant designed to help blind users understand their environment through AI and computer vision.

What it does

NOVA combines several AI Vision capabilities into one system.

It can:

• Detect objects in the user's surroundings using computer vision.
• Track detected objects and identify their position in the camera view, such as left, center, or right.
• Identify the dominant color of detected objects.
• Read text using OCR in Arabic and English.
• Recognize objects and provide information about what the camera sees.
• Remember objects and their locations so the user can ask about previously seen objects.
• Use depth estimation to understand the distance structure of the environment.
• Describe visual information through text-to-speech, allowing information to be communicated without requiring the user to look at a screen.
• Provide a user interface for interacting with NOVA and its different capabilities.
• Use an AI model such as Llama 3B as part of the system's vision/intelligence pipeline.

For example, NOVA can produce information such as:

“Couch in the center with blue color.”

The goal is to turn raw camera input into information that is useful and understandable to a blind user.

How we built it

NOVA was built as a modular AI Vision system using Python and multiple computer-vision and AI components.

The core system uses YOLOv8 for object detection and tracking. Detected objects are processed to determine their position in the camera frame and their visual characteristics.

A dedicated color-processing component identifies the dominant colors of detected objects.

NOVA also includes a memory system that stores information about detected objects and their locations. This allows the system to move beyond one-frame detection and work with information collected over time.

For text understanding, NOVA uses OCR capabilities to read Arabic and English text from images.

We also developed a depth-estimation component using MiDaS to provide information about the spatial structure of the scene.

The system uses text-to-speech to convert the visual information into spoken feedback. Processing is designed to avoid constantly repeating the same information, making the audio output more practical.

NOVA also has a dedicated UI that brings the system's capabilities together instead of treating every component as a separate experiment.

Challenges we ran into

One of the biggest challenges was combining multiple computer-vision capabilities into a single system instead of building isolated demos.

Object detection, tracking, color recognition, OCR, memory, depth estimation, AI reasoning, and voice output all have different requirements. Making them work together while keeping the system responsive required a modular architecture.

Another challenge was deciding what information should actually be communicated to the user. A vision system can detect many objects at once, but continuously describing everything would create too much audio information.

We also had to deal with the difference between recognizing an object and actually understanding its context. Detecting “couch” is useful, but knowing that it is in the center, blue, and previously seen in a particular location provides much more useful information.

Depth estimation was another challenge because it adds an additional perception layer that has to work alongside the existing vision pipeline.

Accomplishments that we're proud of

We are proud that NOVA has grown from a basic object-detection idea into a multi-component AI Vision system.

The project now combines:

• Object detection and tracking
• Spatial positioning
• Color recognition
• Arabic and English OCR
• Object/location memory
• Depth estimation
• AI-based visual understanding
• Voice output
• A dedicated user interface

We are especially proud of the memory capability, because it changes NOVA from simply describing what the camera sees into a system that can retain information about the environment.

We also built NOVA with accessibility as the main objective rather than treating accessibility as an additional feature.

What we learned

Building NOVA taught us that creating an AI application is not just about choosing a powerful AI model.

We learned how different AI and computer-vision components can be combined into a larger pipeline, and how important system architecture becomes when multiple models and processing stages need to work together.

We also learned more about:

• Real-time computer vision
• Object detection and tracking
• OCR
• Depth estimation
• Visual feature extraction
• Spatial memory
• AI-assisted visual understanding
• Text-to-speech interaction
• Designing AI systems around a real user's needs

Most importantly, we learned that an AI system becomes much more useful when it converts technical model outputs into information that a person can actually use.

What's next for NOVA

Our next goal is to make NOVA more capable of understanding the user's environment as a complete scene rather than as a collection of independent detections.

We want to improve its spatial memory, contextual understanding, and interaction with the user.

Future development can include:

• Stronger spatial and environmental memory
• Better scene understanding
• More natural voice interaction
• Improved depth and distance understanding
• More efficient real-time processing
• Better contextual reasoning
• More reliable testing in real-world environments
• Further improvements to the NOVA interface and user experience

The long-term goal is for NOVA to become a practical AI Vision assistant that can help blind users understand their surroundings and interact with the world with greater independence.

Built With

• computer-vision
• machine-learning
• ocr-it
• ttsengine
• yolov8