Photo Screen: Select a picture source
Results Screen: Read the text here
Settings Screen: Customize!
I was inspired to create this app by my aging Chinese grandparents, who are trying to learn English by reading books. I noticed them struggle with certain words and lean in squinting to read the next paragraph. I decided that I could create a better way.
What it does
Spectacle is an upgrade for your reading glasses. Through OCR, you can take a picture (or choose from your camera roll) of any text in any language, be it book, newspaper, or otherwise, and convert it into accessible text (everything is in Verdana, the most readable screen font). In addition, through English word frequency analysis, we link definitions to the harder words in the text so you can follow along. Everything is customizable: the font size, the difficulty of the text, and even the language the text translates to. Being able to switch back to native Traditional Chinese to solidify their understanding of the text is a blessing for English language learners like my grandparents.
With this list of features:
- Translate images to any language
- From any language
- English word difficulty detector and easy-to-access definitions
- Replace hard words with easier ones inline.
- Accessibility: font and font size
Spectacle can help younger people and non-native English speakers learn English vocabulary as well as help older/near-sighted people read without straining their eyes.
How I built it
Spectacle is built on top of Expo.io, a convenient framework for coding in React-native yet supporting iOS and Android alike. I decided this would be best as a mobile app because people love to read everywhere, so Expo was definitely a good choice. I used various Google Cloud ML services, including vision and NLP, to render and process the text. Additionally, I used Google Cloud Translate to translate text into other languages. For word frequencies and definitions, I combined the Words.ai API with my own algorithm to determine which words were considered "difficult".
Challenges I ran into
Although I had used Expo.io a little in the past, this was my first big project with the framework, so it was challenging to go through the documentation and see what React-native features were supported and what weren't. The same can be said for the Google Cloud Platform. Before this, I had only deployed a Node.js app to the Google App Engine, so getting into it and using all these APIs was definitely tough.
Accomplishments that I'm proud of
I'm proud that I got through the challenges I listed above and made a beautiful app (along with my team of course) that I will be proud to show to my grandparents. I'm also proud that I set lofty, yet realistic goals for the app and managed to meet them. Most of the time, when my team goes to a hackathon, we end up trying to add too many features and have an unfinished product by the time it's over, so I'm very glad we didn't let it happen this time.
What I learned
I learned a lot about Google Cloud Platform, Expo.io, and React-native, as well as how to put them together in (maybe not the best) but a working way.
What's next for Spectacle
I want to add the ability to save images/text for later, so that you can essentially store some reading material for later, and pull it up whenever you want. I also want to further upgrade the hard word detection algorithm that I made.