As language learners, it's important to constantly be using the target language that you wish to learn. LinguaLens gives users the experience of being immersed in their target language as they go about their day.
Games like PokemonGO served as our inspiration for developing an immersive experience combining language learning with a fun twist.
What it does
LinguaLens allows users to take a picture of any object using their phone. The app will then use Clarifai's visual search to detect the objects in the photo and return a word in the target language.
How we built it
Using React Native and the Expo framework, we were able to quickly create a prototype of LinguaLens. Using React Native was critical for us because it significantly reduced the amount of time it took us to load, test, and modify the app. We used Expo's Camera API to take pictures through the camera and the Clarifai API to predict what objects the user is seeing. From the Clarifai API, we were able to display to the user the names of the object in the user's desired language to learn.
Challenges we ran into
Problems encountered included saving pictures taken with the camera and communicating with the Clarifai API. One of the most difficult tasks was deciding how to structure the program and its various subcomponents.
Accomplishments that we're proud of
We're proud to have integrated the Clarifai API with the user camera to deliver a comfortable experience in learning another language.
What we learned
We learned the difficulties that come with joining an external API with a base feature of mobile phones. We also discovered that the design process requires a lot of time and devotion before one can start diving into the code.
What's next for LinguaLens
We plan to develop LinguaLens into a fully functional game, similar to PokemonGO. Users will have accounts that allow them to store and manage the words that they have previously learned. Additionally, we would provide forums for the users to interact with eachother and to aid each other. The game would also have different levels and challenges, each level would unlock different attributes. For example, level one would help users learn nouns, while level two would help users learn adjectives.