Braille reader with teaching features

Inspiration

We were taken aback by the lack of accessibility for the newly blind on the Internet and in the physical world. One particular example that stood out to us was the cost of Braille-readers - starting from a few hundred pounds through to over a thousand, we felt that they were simply too costly to be accessible to most people who needed them. On top of this, there were no good tools online to effectively translate Braille into other languages - only one attempt was made, but it resulted in illegible text that was nowhere near good enough for practical use. This is why we decided to put together a system using generative AI to read and translate Braille into other languages, while also helping users learn Braille through this method. It was another way in which we could support people in connecting to the rest of the world.

What it does

It does a number of things, including reading Braille from images of real printed Braille, translating it into natural languages, teaching and testing learners on their comprehension.

How it works

The backend consists of a first API call to Angelina, a Braille image to Roman letter converter that attempts to translate every character, but with a few mistakes. Then this output is scraped and fed into Claude for imputation of the missing characters. We then fed it back through a pipeline to frontend interfaces, that used this data to help users learn Braille and assist them with their reading.

Accomplishments that we're proud of

We spent a good amount of time planning how to approach our solution, eliminating many problems before they could significantly hold us back. We worked successfully as a team and made rapid progress on our application, making use of strategies such as task delegation and paired programming, successfully integrating our parts of the application together so that they worked synchronously and harmoniously. We successfully put together a chain of services to convert Braille images into natural language text and audio, still maintaining fast response times throughout the pipeline. Finally, we persevered through problems, even if they took hours to solve - this was achieved with a combination of strong teamwork and deep research on how best to work around issues.

Challenges we ran into

Initially, we took a long time in deciding how exactly to solve our problem. Simply translating Braille into English is not very helpful to the visually impaired, who usually require it the other way round! We considered translating it the other way round, but this required building very specialised hardware, which was not feasible to build and test in the time available. This was when we had the idea of designing our application in such a way that it helped people to learn Braille. Concerning hardware, the first raspberry pi was faulty, and connecting to networks and bluetooth speakers was a challenge, but certainly an enjoyable one. Connecting parts of the pipeline together was one of the most difficult challenges we faced, due to having to design and adjust components to work seamlessly with each other - however, we achieved this through our work as a team.

What we learned

We learned to understand technology from the perspective of a much more diverse range of users, having tested our application meant to help blind people. It reshuffled our priorities greatly - accessibility became so much more important than an attractive User Interface, and the use of sound became much more important.

What's next for Braille reader with teaching features

There is plenty of room for further improvement and polishing of our application. Some of these improvements are in our frontends, including more accurate speech recognition, better accuracy comparisons and more flexibility for learners of differing proficiency. Some of those in our backends include experimenting with implementing our own Braille character recognition system, and supporting many more languages, including those with a very different writing system to English. The latter of these two is one that has massive future prospects and offers a good opportunity to greatly improve and grow our application.

Built With

• angelina
• anthropic
• api
• claude
• django
• elevenlabs
• git
• gpiozero
• javascript
• opencv
• python
• raspberry-pi
• react
• webcam