Inspiration

We came across a person in our organisation named Mr Dinesh Kaushal who is a senior developer in our company Publicis Sapient. Although he is blind ,he together with his wife manages to code and debug perfectly by using Microsoft and other Narrators. But still he faces many day to day challenges one of which is Reading emails which are image based .

What it does

We have used Google vision and UiPath capabilities to extract the Image Context as well as it's text fields out for him which is then given as a speech output making his life simpler and easier now.

How We built it

We have asked Dinesh to share his mail list which he has rejected due to narrator lacking capabilities. and have worked together on those mails with my team to create a solution using Google Vision.

Challenges We ran into

There are many types of emails which we are not aware off and take for granted . Mails with links , mails with link embedded on photo, mails with attachments hidden, mails with native text on text boxes etc . It was a very challenging task to create a flow which can takes care of all the mailing variations. The current solution created handles all these variations .

Another challenge was to manage speed of the speech for Dinesh . We needed to create a speed controller which can help him in listening the mails properly .

Accomplishments that we proud of

Although my Team won UiPath's Gurgaon Hackathon . But one think that really makes me proud is Dinesh life is much easier now. He personally has given appreciation to our manager and has raised it to IT department to use this company wide in our organization .

What we learned

We learned about Google vision, OCR, Parallel activities launch , hotkey trigger and many more functionalities of UiPath.

What's next for Visual Assistant for blind

*** Integration with Narrator – If our current solution can be integrated with laptops default narrator then it will be very easy for Dinesh to switch things up in single go . Advanced Speed Controller – We need a advanced controller, as different blind people use different controlling speed of narrator.

Geolocation Navigator- As Dinesh is blind it’s very hard for him to navigate within company premises , we wishto use companies's networks like routers, RFID , sensors with uipath that will help him to navigate correctly within company premises where Google map fails.

Integration with Alexa- To make it a ease for blinds when working or at home.

Built With

Share this project:
×

Updates