JumboVision

• 

  Front end application view

WE ARE IN THE BEGINNERS TRACK

Inspiration

We wanted to create something to help the visually impaired. We found that there was a lack of tools online helping to guide the visually impaired and better navigate their surroundings. We wanted to improve the safety of the visually impaired and assist them in living a more fulfilled life.

What it does

Provides text to speech descriptions of what the users camera frame is pointing at, to aid in navigation. Describes surrounding in concise, but clear detail.

How we built it

We used a react app to create the front end application, and connected the camera to a YOLOv8n pre trained AI model, where we sent it frames, and received back text object identification and location descriptions. We we are able to build a connection between the front end and the backend using a WebSocket.

Challenges we ran into

We found it difficult to connect the front end to the backend, as well as creating mobile functionality. We also struggled to deploy our site on Vercel, making it accessible to anyone on the web.

Accomplishments that we're proud of

We are very proud of successfully connecting the backend to the frontend and getting the camera to correctly identify objects it can view. We knew this would be the hardest part but we were able to get it done.

What we learned

We learned a lot, including how to use websockets and how to better use React, Tailwind, YOLOv8n, Hugging face models, python, fastAPI, webSockets for streaming real time video, and virtual enviroments.

What's next for JumboVision

We hope to add better mobile support in the future. We also hope to have the user switch between front and back cameras, and maybe even incorporate a 360 camera. We also want to hopefully add voice command features for the users.

Built With

• css
• javascript
• python
• react
• tailwind
• typescript
• yolov8n