SpAnki

Inspiration

After seeing the Sophomore's Social Justice project in Devlin Hall, we began thinking about the ways technology could help immigrants arriving in the U.S. Recognizing the difficulty to learn English, we deciding to make a technology that can help immigrants learn English they see all around them in the U.S.

What it does

With the SpAnki app, the users take images with their phone, and draw boxes around any word they do not know. The bounding boxes are then sent to our server, where tesseract determines what the word is. The word is then stored in FireStore, and a python script queries the word. Using Merriam Webster and Oxford Dictionary APIs, the Spanish equivalent of the word and its English definition is found. Then using parsing and AnkiConnect, an Anki plugin, we autogenerate flashcards for the user.

How we built it

We used Swift to develop the app, which mostly consists of a camera interface and the ability to draw bounding boxes around any part of the image. When a bounding box is created, it sends the data to the server. Then using OCR on the server, the English word is identified. Python queries FireStore and the APIs, and Python was used to interact with Anki's API.

Challenges we ran into

A challenge we faced was making bounding boxes for the users. Figuring out how to enable the user to draw the boxes easily and capture the 4 coordinates of the box to enable OCR was difficult. This was because the image coordinates and frame coordinates were not the same, so we had to correlate the two.

Accomplishments that we're proud of

We're proud to have implemented a fairly sleek Swift camera interface that enables users to identify the words they do not know. We also are happy that we can correlate the user's account on the SpAnki app to the Anki app, so that each user's data is synched across both apps. The flashcard update happens all behind the scenes, so the user does not have to manually do anything except put a box around the words in the SpAnki app.

What we learned

We learned about Anki plugins, how to make a camera in swift, how to query FireStore DB, and how to use Tampermonkey to alter the buttons on google docs.

What's next for SpAnki

We hope to add correction to the OCR, because right now the OCR does not utilize human in the loop to ensure a word was not misinterpreted. We would also add CronJob to our python script, so it.

Built With

• ankiconnect
• firestore
• javascript
• merriamwebsterapi
• oxford-english-dictionary
• python
• swift
• tampermonkey
• tesseract