Inspiration

There are millions of patients with dyslexia in the United States and across the world. Dyslexia is a particularly challenging learning disability for patients in elementary school - struggles with reading, math and learning sequences makes it very challenging to obtain a basic scholastic foundation.

As individuals with a profound interest in medicine and the healthcare technology space, we hoped to offer a solution to a significant yet unaddressed issue. After having met a patient with dyslexia recently, one of our members realized how significant of an impact it had on students. While this patient was a very bright individual, he struggled in school due to his reading level, challenges with learning sequences and understanding the order of mathematical symbols, and having troubles adjusting due to his difficulties with reading large paragraphs/books. We wanted to offer a solution which distills the task of reading to its basic components and which removes distractors and visual stresses to make reading a seamless and truly educational process.

What it does

The user uploads an image of the text they would like to read and using Python, pytesseract and Tesseract-OCR, this image is processed and converted into text which is then fed to the WebApp. The WebApp displays the text in easily managed chunks and highlights the current word (almost like a cursor) to facilitate learning. The user can read along as the cursor moves, and words which are challenging can be added to their personal list to be referred to at a later time. Furthermore, the user can view definitions and pronunciations to make their learning more meaningful. The number of difficult words and a Flesch-Kincaid Grade Level score are provided to help users get a gauge on how challenging the particular text they are reading is. Words which are difficult will also be identified to aid in the user's comprehension.

How we built it

The image which the user uploads to the WebApp is processed using Python, pytesseract, Tesseract-OCR and the textstat Python package to yield a text string which is handed over to the WebApp via flask and HTTP. A Flesch-Kincaid score, a set of difficult words, and the number of difficult words is also calculated using the textstat package and a custom function. We built the WebApp in HTML using JavaScript and more specifically the react.js framework. Speech-to-text was provided by webkit-speech-recognition and text-to-speech was provided by webkit-speech-synthesis.

Challenges we ran into

We wanted to make sure that the website we designed was accessible to dyslexics, so we consulted the British Dyslexia Association's Style Guide to ensure that fonts, spacings, colors and overall design were suitable for patients with dyslexia. This was a time-consuming but ultimately worthwhile process, as it improved the look and feel of our WebApp. It was also initially challenging to ferry data between the Python backend and the WebApp frontend, particularly with the transfer of images. This was resolved using HTTP and flask.

Accomplishments that we're proud of

We are proud that the WebApp can transcribe almost all of the images we submitted with few errors, and we hope that our users will gain something from our work.

What we learned

We learned how to apply speech-to-text, text-to-speech and image-to-text capabilities to a serious yet unaddressed problem.

What's next for Clarify

We hope to add additional functionality to break down math equations into more meaningful pieces, such as numbers, operators, and exponents. We would also like to offer our service to patients with dyslexia to evaluate how we can improve the user experience and fine-tune any issues with the OCR.

Built With

Share this project:

Updates