Aria Vision

Inspiration

Understanding your surroundings can be challenging, especially for people who are blind or visually impaired. Everyday activities, such as recognizing objects, understanding what is nearby, or getting oriented in an unfamiliar place, can require extra effort.

We wanted to explore how AI and wearable technology could make this experience easier. That inspired us to build AriaVision, an AI-powered assistant that uses Meta Aria glasses to help users understand their surroundings through spoken descriptions and answers to their questions.

Our goal was to make interacting with the world more accessible by letting users get information through audio instead of constantly looking at a screen.

What It Does

AriaVision connects Meta Aria glasses with an AI assistant that helps users better understand what is around them.

With AriaVision, users can:

Receive spoken descriptions of their surroundings.
Press the Space key to ask questions, such as “What is in front of me?”
Ask for help understanding their environment or figuring out where to go.
Hear the assistant's responses instead of needing to read them on a screen.

The project explores how wearable cameras and AI can work together to provide more accessible, hands-free environmental assistance.

How We Built It

We combined wearable hardware, AI, and software to create our application.

Meta Aria glasses: Capture visual information from the user's surroundings.
Gemini: Powers the AI experience by interpreting visual information and helping generate responses to questions.
TypeScript, HTML, and CSS: Support the website's interface and interactive features.
Python: Runs a bridge that connects the glasses to the application and transfers camera information.
Project Aria Client SDK: Helps the application communicate with the Meta Aria glasses.
WebSockets: Allow the Python bridge and website to exchange information.

We also worked on connecting the glasses to a computer through USB so the application could receive information from the device. Our goal was to combine this visual input with spoken responses to make the experience simple and intuitive.

Challenges We Ran Into

One of our biggest challenges was getting the hardware and software to communicate properly. Connecting the Meta Aria glasses required setting up the Python environment, installing dependencies, and troubleshooting the USB connection.

We also ran into network restrictions that interfered with our wireless connection attempt. Using USB gave us another way to connect the glasses to the computer. Additionally, a port conflict on the computer required us to choose a different port for communication.

Another challenge was bringing the different parts of the experience together. We wanted the assistant to provide environmental descriptions while also allowing users to ask their own questions. Working through these challenges taught us how much coordination is required when combining AI software with wearable hardware.

Accomplishments We're Proud Of

We're proud to have explored how Meta Aria glasses can be used as part of an AI-powered accessibility tool. We connected the glasses to our application, created an interface for user questions, and developed spoken environmental guidance.

We're also proud of the idea behind AriaVision: making information about the surrounding environment easier to access through audio. This project let us explore how AI can go beyond a traditional screen-based experience and become part of a wearable, real-world interaction.

What We Learned

Throughout this project, we learned that building an AI application involves more than simply connecting a model to a website. Hardware integration, device communication, network limitations, and the user experience all play important roles.

We also learned how important it is to design with accessibility in mind. An assistant is more useful when its information is easy to understand and doesn't require constant attention to a screen.

Most importantly, we learned the value of troubleshooting, adapting when our original approach did not work, and testing individual parts of a system before bringing them together.

What's Next for AriaVision

AriaVision still has room to improve. In the future, we would like to:

Improve the accuracy and usefulness of environmental descriptions.
Expand the types of questions the assistant can answer.
Explore more advanced navigation assistance.
Improve reliability across different network and connection setups.
Test the complete experience more extensively with the physical glasses.
Gather feedback from people who could benefit from hands-free environmental assistance.

Our long-term vision is to make AriaVision a more reliable and useful accessibility companion that helps people better understand their surroundings and move through unfamiliar environments with greater confidence.