AURA

• 

  Future Glasses with inbuilt speakers

• 

  Future Glasses

• 

  HomePage of Aura

• 

  setup

• 

  Guardian Setup

Inspiration

I wanted to explore how a voice-first assistant could make everyday visual tasks easier for people who are blind or have low vision. Finding familiar items or understanding what is nearby can take extra effort when information is presented mainly on a screen. AURA is built around a simple idea: ask aloud, then hear a useful response.

What it does

AURA is a browser-based visual companion prototype. Users can ask it to describe a camera view or look for an everyday item, then hear its response. It is designed for hands-free interaction after the user grants microphone and camera permissions.

The interface also includes a heart-rate and SOS demonstration to explore how wearable data could support a guardian experience. The readings in that demo are simulated; live guardian sharing and emergency alerts are not connected in this version.

How we built it

AURA combines a web interface with a local Python server. Local Whisper handles speech transcription, while Ollama runs the vision model on the laptop. For a visual request, AURA sends a camera frame to the local model for analysis.

I also created a 3D concept image of smart glasses with a camera in the frame and speakers near the ears. It illustrates a possible future design; the glasses are not built or connected. For this hackathon, AURA is a simulated Alexa+ experience in a web app, not an Echo or Alexa service integration.

Challenges we ran into

Speech recognition and response time vary with the computer, microphone, model, and language. The vision model can also confuse similar-looking objects. Wearable data brings another challenge: watches differ in what they expose to a browser, and sharing health or location data with guardians requires a secure service and user consent.

Accomplishments that we're proud of

We brought voice input, camera-based assistance, local AI processing, and spoken responses together in one working prototype. AURA can demonstrate a spoken scene-description flow and an object-search interaction. We are also proud that the project presents its wearable and glasses ideas as future possibilities rather than claiming hardware that does not exist yet.

What we learned

We learned that an accessible assistant needs more than image recognition. It needs clear spoken responses, a usable hands-free flow, privacy-conscious processing, and honest wording when it is uncertain. We also learned to test language recognition and response time on the actual device instead of assuming they will work equally well for everyone.

What's next for AURA

Next, we want to test AURA with people who have low vision and measure how reliably it understands requests, identifies items, and responds. We also want to evaluate compatible wearable integrations, build a secure opt-in guardian connection, and explore the glasses concept as a separate hardware prototype. Those integrations are future work; the current demo does not provide medical monitoring or emergency dispatch.

Built With

• accessibility
• ai
• alexa+
• api
• bluetooth
• computer
• css
• gps
• html
• hugging-face
• javascript
• local
• mediadevices
• ollama
• python
• pytorch
• qwen2.5-vl
• recognition
• speech
• transformers
• vision
• web
• whisper