SensAi

Inspiration

To benefit the visually impaired, we simulate the vision of a blind person by using a mobile device as the camera to see and process data from the environment.

What it does

The app leverages computer vision to recognize objects in the person's path, guiding them around detected obstacles using voice commands. It also allows users to scan text, which is then read out loud to them.

How we built it

Frontend : We started the project using a template from a previously built website by a team member. The project has a ninja-themed design. The color scheme is red, black, and white, and we chose a logo from a Canva template that matched. We used the Berlin San FB font, which complemented the logo design. The logo was removed from the webpage header for stylistic reasons. We swapped out old images with a forest image to align with the ninja theme. We also adjusted margins, font size, alignment, padding, and width. Finally, we added a copyright notice and a ninja icon at the bottom of the page.

Backend : The app prompts users to log in for access. Once logged in, it provides webcam access. Text recognition (OCR) is implemented using Tesseract.js. Object detection is handled using TensorFlow's Coco-SSD model. Voice recognition is achieved using the Web Speech API. Text-to-speech functionality is provided by the Web Speech API. Voice command handling is integrated.

Challenges we ran into

We faced issues with detecting incoming movement. Object location detection was problematic due to incorrect center screen values, causing objects to be misplaced. We had difficulty getting the correct audio output for the selected mode. Proper webpage alignment was a challenge. Connecting the API to a mobile device presented some issues.

Accomplishments that we're proud of

We are proud to have successfully made a product that works on mobile devices. Utilized TensorFlow, Tesseract, and React we created a fully functioning computer vision project. Furthermore, we are proud of the website's design.

What we learned

Being particular about our code and saving previous versions is helpful for troubleshooting. Time management is crucial. More styling does not necessarily mean better results. Having a set theme early on in the challenge proved to be useful.

What's next for SensAi

We plan to transition the product into a wearable piece of hardware.

Built With

• css
• javascript
• react
• tensorflow
• tesseract.js