Augmented Reality Interactive Language Learning

Canada alone, annually 300,000 people immigrate here. A substantial sum of these immigrants don't know the main language of the country their moving to. We set out to make an interactive AR app where you can learn by seeing familiar objects, and touching them in AR! We first intended to make this app on Xcode, EchoAR, and ManoMotion but we ran into issues with loading our Echo AR models into Xcode. We made models using blender. We shifted out focus towards semantic segmentation with pytorch and a U-NET model. We wanted to do this to create a mask of a persons hand so you can easily use an if statement and detect if you touched the model in AR. Again we ran into problems with pytorch. Lastly, we tried making a mask by converting colors in a webcam feed with opencv. We did end up making a succeful mask. Since we couldn't use Xcode at this point we had to find a way to integrate AR into python. We made a sort of comprimise by making a program which turns the background of your webcam feed to a video of your choice. Technically it was still AR but we couldn't do the touch thing. The good thing about this is you can put it in your phone and have an immersive experience, but we didn't have time for that. So we changed the mask to the next best thing, hand gesture detection. In this specific lesson the object that was being projected was a pencil and the question(in chinese) was: is this a toothbrush? Holding up 1 finger meant no and holding up 2 meant yes. If you show the correct answer your screen will show a text either, Correct! or Wrong :|(or put your hand in the box if you haven't done so). In this project we learned more about echo AR, pytorch, and opencv.

Built With

• numpy
• opencv
• python