Prism

• 

  An image showing the description of the project, the project logo, and screenshots of the app.

• 

  The Prism logo.

Inspiration

While on the plane several weeks ago, I witnessed a mom helping her child onto the plane. She was describing everything to him as he walked, including obstacles and things she was seeing along the way. This inspired us to build an accessible solution that lets people with vision impairments be able to navigate through difficult situations with multiple obstacles easily.

What it does

Prism is an iPhone app that connects to your Meta Ray Ban glasses and helps you with navigation and obstacle avoidance while walking. We utilize multiple AI models running both on-device and on AMD servers with text to speech provided by ElevenLabs.

How we built it

We used Swift for the app and python for the VLLM backend.

Challenges we ran into

• We had issues when training the YOLO model because Winograd was not working correctly on AMD GPUs. We disabled the Winograd convolution layer which fixed our model training.

Accomplishments that we're proud of

• Trained our own model on AMD infra on our collected and labeled data
• Gathered and labeled custom data for our own model
• Implemented Meta's brand new Wearables SDK with multiple models
• Built a VLLM backend on AMD developer cloud

What we learned

• How to build accessible design
• Training models on AMD infra
• Swift apps with streaming video data

What's next for Prism

Improving our models with more data and training.

Built With

• amd
• python
• qwen
• swift
• vllm
• yolo