Empowering the visually impaired using AI

Inspiration

In the current days of innovation, we have tools to read epapers/articles aloud to the visually impaired folks. But they still cant gain any information about the graph/images in the articles. We aimed at building a tool to help them gain insight/summarize graph and images using Artificial Intelligence.

What it does

Provides the visually impaired with speech and sound outputs, for interpretation of graphs. -> By employing CNN, we successfully developed an AI for classification of graphs. -> We are able to classify mathematical graphs 95s% accuracy with a limited training DataSet.

How we built it

We generated different mathematical functions like sin, logarithmic, exponential, linear etc and then used machine learning techniques to classify the graphs of these functions to their underlying mathematical function. Thus we assume that we don't have access to the underlying data and just have access to the graphs which we classify into different functions.

We experimented with many different classifiers like support vector machines and convolutional neural networks and obtained around 95% accuracies with both of them.

Due to lack of time, we could not try this kind of classification to classify different types of charts like line, area, histogram, pie charts etc but our approach can be easily extended to classify this kind of charts as well.

Given the output of the classifier for a new test graph, we then used Microsoft speech API to generate descriptive speech output that can be well understood by visually impaired people.

Challenges we ran into

• Sound Representation: creation of sound notes to represent mathematical functions was unable in any existing libraries. • Accuracy: Classification of graphs by employing statistical machine learning techniques proved to be inaccurate. Hence we were motivated to build our own convolutional neural networks model to achieve feasible results. • Text to speech conversion which we initially perceived as a challenge was later simplified thanks to a brilliant Microsoft API. • Final integration of the complex AI functions into a Django (Cherrypy) web application was challenging.

Accomplishments that we're proud of

• Empowering the visually impaired using our application, is our greatest achievement: Using our application, the visually impaired can gain insight into mathematical graphs and functions which was previously not possible. • Creation of a highly accurate Convolutiional Neural Networks to identify mathematical graphs with a very high degree of accuracy (95% - with a limited data set).

What we learned

• Superiority of Neural networks over statistical Machine Learning for certain tasks. • Functionality of Microsoft's Text to Speech API and Computer vision API

What's next for Empowering the visually impaired using AI

• Integration of the webapp into popular daily products. Such as plugins in browsers, e-readers, tools in newspapers and other graph related content magazines. • Developing the software into a deployable/marketable product.