NOVA

NOVA is an AI-powered vision assistant designed to help blind people.

Comment

Inspiration

NOVA was inspired by a simple question:

What if technology could help a blind person understand their surroundings in a way that feels more natural than simply detecting objects?

We wanted to build something that goes beyond saying “person,” “chair,” or “bottle.” Our goal was to create a vision system that can understand what is happening around the user and communicate useful information through voice.

The idea came from thinking about everyday situations where visual information is important: identifying objects, understanding where they are, recognizing their colors, reading text, and remembering where something was seen.

That led us to NOVA — a vision-based assistant designed to turn visual information into useful, accessible interaction.

What it does

NOVA is a computer-vision assistant that processes visual information and turns it into spoken information.

Its current vision pipeline combines several capabilities:

  • Object detection and tracking using YOLO.
  • Object positioning to determine whether an object is on the left, center, or right of the scene.
  • Color recognition for detected objects.
  • Visual memory, allowing observations to be stored and recalled.
  • Depth estimation using MiDaS to understand the spatial structure of a scene.
  • Voice output using text-to-speech so that visual information can be communicated without relying on a screen.

The system can process a live camera feed, while recorded video is also used during development to repeatedly test the same scenarios.

NOVA is designed as a modular system, so these different capabilities can work together rather than functioning as completely separate AI experiments.

How we built it

We built NOVA as a modular Python-based computer-vision system.

The main pipeline uses YOLOv8 for object detection and tracking. Detected objects are analyzed to determine their position in the frame, while a separate color-processing component analyzes the dominant colors inside the detected object's bounding box.

We implemented a memory component that stores visual observations and can later be queried by the user. The current prototype already supports asking the memory system questions such as where an object was seen.

For depth estimation, we developed a separate MiDaS-based pipeline. It uses MiDaS_small, selects CUDA when a compatible GPU is available, and otherwise runs on the CPU.

The system also uses text-to-speech to communicate visual observations. We run speech generation asynchronously so that speaking does not stop the visual processing loop.

During development, we used recorded video as a repeatable testing environment. This made it easier to test and debug the computer-vision pipeline before relying entirely on a live camera.

Challenges we ran into

One of our biggest challenges was making several computer-vision components work together while maintaining real-time performance.

Object detection, tracking, color analysis, depth estimation, memory, and speech all have different computational requirements. Combining them into one system required us to think carefully about how the pipeline should be structured.

Another challenge was that detecting something is not the same as understanding it.

A model can tell us that an object exists, but a useful assistant needs additional context: where the object is, what it looks like, and what other information has been observed about it.

Depth estimation was another challenge. We initially developed it as an independent test so that we could verify that the MiDaS model worked correctly before integrating it into the larger NOVA pipeline.

We also had to think about audio interaction. A system that speaks every time it detects an object would quickly become overwhelming, so NOVA includes mechanisms for controlling when new information should be spoken.

Accomplishments that we're proud of

We are proud that NOVA grew from an idea into a working computer-vision system composed of multiple AI and software components.

We successfully built a pipeline capable of combining:

vision → detection → tracking → color understanding → memory → voice

and separately developed a working depth-estimation component.

We are especially proud of the modular architecture. Instead of creating one large piece of code, we separated capabilities such as color analysis and memory into their own components, making the system easier to test and extend.

We are also proud of building NOVA around an accessibility problem rather than building AI simply as a technical demonstration. Every component is intended to contribute toward helping a user understand visual information.

What we learned

NOVA taught us that building an AI project is very different from simply using an AI model.

We learned how to build and connect computer-vision components, work with real-time video streams, use object tracking, process images, perform depth estimation, manage model inference, and integrate voice output.

More importantly, we learned that system design matters as much as individual models.

A highly accurate model is not enough if the surrounding system cannot combine its output into useful information. We learned to think about the complete pipeline: what the camera sees, what the AI understands, what information should be remembered, and what the user actually needs to hear.

We also learned the importance of testing individual components separately before integrating them into the complete system.

What's next for NOVA

Our next goal is to turn the current collection of vision capabilities into a more unified context-aware vision assistant.

We want NOVA to combine object detection, depth, memory, and user intent into a shared understanding of the scene.

Future development will focus on:

  • Integrating depth information directly into the main vision pipeline.
  • Improving spatial memory so NOVA can better remember where objects were seen.
  • Adding more natural user interaction.
  • Improving scene understanding and contextual reasoning.
  • Making NOVA faster and more efficient on everyday hardware.
  • Continuing to improve the interface and overall user experience.
  • Testing the system in more realistic environments instead of controlled demonstrations.

Ultimately, we want NOVA to move from “I can detect what is in front of me” toward “I understand what is happening around me and can tell the user what actually matters.”

Built With

Share this project:

Updates

Submission history