Languages are hard to learn and pronunciation is often especially difficult, which all of us had experienced first-hand. We decided to create a real-time augmented reality language learning game called Ceci ("this" in French, pronounced as say-see).
What it does
Ceci quizzes the user on the vocabulary of the language they are studying based on what they see in the world. It highlights the word they are being quizzed on with a box around the corresponding object and recognizes it with machine learning. The user says the word, and Ceci uses voice recognition to detect whether or not they are correct. To incentivize the user, there is also a point system.
How we built it
Using CoreML for machine learning, Ceci is able to detect and label possible objects to quiz the user on. Then, we used the built-in Xcode speech recognition tool to check the user's answers. In general, everything was written in Swift, including the point system that rewards correct answers.
Challenges we ran into
We initially planned to use many ARKit features, but quickly discovered that the quality of the classification in its object detection is lacking. Object detection is central to Ceci, so we were forced to find something else. Instead, we used another machine learning library, and it was a bit of a challenge to go through the non-documented issues and limitations, due to the relative novelty of this technology.
Accomplishments that we're proud of
We are proud that we were able to combine various exciting technologies into Ceci. For example, we used a scalable, mobile machine learning library that none of us have ever used before, and incorporated it along an Apple-developed speech-to-text transcription.
What we learned
Most of the team wasn't familiar with Swift specifically and iOS development in general, and learned them to develop features like the points system. None of us had done iOS augmented reality before so we had to experiment with a lot of platforms and ideas to decide what was feasible. Also, most of the team didn't know most of the others when we started, so we learned how to work together most efficiently and to leverage our strengths.
What's next for Ceci
We intend to and can pretty easily add more languages to Ceci such as German, Spanish, Russian, and Chinese (including Mandarin and Cantonese). We also want to make Ceci more social, adding support for sharing words learned and a leader board. In addition, building on the point system to make points redeemable for custom themes and improving the choice of quiz objects based on spaced repetition learning are major features we hope to implement.