WordLens AR

Using computer vision, ML, optical character recognition, and AR, this iOS app serves as an assistive technology to aid dyslexic individuals in enhancing their reading and comprehension skills.

Comment

Inspiration

As an avid reader, one of my favorite pastimes is diving into a good book. More than just a hobby, reading has enabled me to improve my critical thinking and emotional intelligence. Discovering the formidable challenges and educational inequality faced by dyslexic individuals in reading deeply moved me, compelling me to take action. Eager to make a meaningful impact, I became motivated to aid dyslexic individuals in achieving fluency in reading and writing. As such, I created WordLens AR to support dyslexic individuals in overcoming learning challenges and promote social equality in schools.

What it does

Dyslexia, the world's most common learning disability, presents difficulties in spelling, writing, and reading comprehension due to struggles in connecting letters to sounds. This condition, unrelated to intelligence but a result of neural hardwiring, unfortunately carries stigma concerning dyslexic individuals' intellect.

To address these challenges, I developed WordLens AR, an innovative iOS app that utilizes machine learning for word recognition and augmented reality to aid dyslexic individuals in reading, writing, and visualizing their progress.

The app's user-friendly interface includes directions for its three features. I optimized the UI for smooth dyslexic user experience, incorporating gif images for demonstration and speech buttons for auditory directions. Face ID facilitates easy access.

The first feature, reading assistance, allows users to resize and position the scanning box over the text they want help with. By clicking the speech button, the app's machine learning algorithm processes the text through optical character recognition, then reads it aloud using text-to-speech technology. Alternatively, the AR button activates an augmented reality model of the word, enhancing comprehension for dyslexic users.

The second feature, writing assistance, addresses spelling challenges with an autocorrect algorithm. The user resizes the scanning box while writing, then presses scan to display possible word options using optical character recognition and a custom autocorrect model. The chosen word is projected in augmented reality onto a detected surface, allowing the user to trace and practice writing it, aiding muscle memory development.

The third feature, progress analysis, provides helpful statistics on most misspelled words and weekly progress in word practice, securely stored on Cloud Firestore.

With WordLens AR, I strive to alleviate some of the obstacles dyslexic individuals face, enhancing their ability to read and write fluently and ultimately supporting their journey in overcoming dyslexia's challenges.

How we built it

Using XCode and Swift, I crafted the core iOS app, integrating UIkit for a simple user interface, SwiftUI for advanced animated UI, and ARkit to showcase augmented reality models from echo3D. The machine learning model, designed for text recognition and classification, employs a cutting-edge OCR architecture, involving a convolutional neural network and an LSTM, followed by CTC with beam search for character extraction. Mindful of dyslexic users, the UI avoids tiny text, utilizing animated images and speech buttons through the AVFoundation framework. To track reading/writing progress, I securely store data on a Cloud Firestore database via Google Cloud. Moreover, I implemented an efficient auto-suggest algorithm in C and Obj-C, utilizing an O(word length) trie and dynamic programming Levenshtein distance calculator for accurate spelling suggestions.

For the UI/UX, I used Figma to design the front-end for my prototype before implementing it into SwiftUI on XCode.

Challenges we ran into

Mastering the augmented reality aspect proved to be quite challenging, but my adept debugging skills came to the rescue, helping me pinpoint and resolve the issues. Likewise, crafting the auto-correct algorithm posed considerable difficulties, and integrating it into the Swift iOS project required persistent problem-solving. Additionally, leveraging machine learning for optical character recognition came with its own set of challenges, particularly when dealing with new data.

Accomplishments that we're proud of

With sincere effort, I aimed to develop an app that genuinely assists those in need. Leveraging cutting-edge tools like ML, OCR, AR, and other advanced coding techniques, I successfully crafted a functional prototype. The hackathon experience proved incredibly enlightening, surpassing education from School and YT vids, and highlighting the significance of hands-on learning.

What we learned

Among the array of tools such as optical character recognition, machine learning, and augmented reality, one stands out as the most crucial: my discovery of creating technology with a purpose, benefiting others in a meaningful way.

What's next for WordLens AR

My aspiration is to continuously enhance the app, refining my machine learning and auto-correct features, with the ultimate goal of publishing it on the app store. I also hope to work with doctors/ophthalmologists to employ this in a clinical lab test to see how it helps dyslexic people.

Share this project:

Updates