Technology has made its way into the classroom in the last couple of decades in many ways shapes and fashions. The addition to hands-on activities through games has made learning more interactive and fun for all. This project hopes to do the same one day with the addition of teaching young children vocabulary using photography or even help someone learn another language through photography. The idea of the project is for a teacher/parent/self to have a mural of many objects, and the game takes that mural and instructs students/users to take a close up picture of specific individual objects within the mural and tell them whether they are correct.

What it does

This algorithm looks at an overall image give, uses the Google Cloud Platform Vision AI to detect object entity names within the overall image. It then looks at other images taken that are close up images of objects within the overall and compare the entity names collected by the Vision AI. If the object detected in the close-up image exists in the mural, then it returns a true value.

How I built it

We coded this project using Java, imported the Google Cloud Platform Libraries needed to make the requests for the entity names and used string comparisons to compare entity names.

Challenges I ran into

We wanted to make the whole game, but our team was not skilled enough in Game Design to figure out how to integrate camera usage on Andriod or Apple Devices.

We also ran into the problem that sometimes the AI gets the name of the object wrong. Solutions to this problem are to have a separate program just detect images and have the creator of the mural/overall image name each object, or to integrate a different neural network that may be able to detect images better.

Accomplishments that I'm proud of

We learned how to install and implement Google Cloud Platform into our IDE and how to your git, something that is not taught at our University. We were able to get this algorithm working in such a way that it could be given to someone who is more skilled in game design and actually developed into a game.

What I learned

How to use Google Cloud Platform and git

What's next for PictoSearch

Take this algorithm back to our University, talk to our head of the Game Design Concentration Degree and see if any of his students would be willing to pick up this project.

Share this project: