Animal Crossing IRL

Inspiration

Before the hackathon, I never made a game before. I really like Animal Crossing and I wanted to try to make an expanded experience of the game that gets people outside and moving around.

What it does

This is a game that uses the camera of your smartphone to scan insects, fish, and works of art and adds them to the Critterpedia and Gallery in the game. When you scan an item, my AI determines what it is and gives you information about it. There is a map that helps you find places where a potential scan can be found, like art structures around my campus and works of art, hung up at the local gallery.

How we built it

After designing the app, we used C# and Unity to program the game. This was a challenge for me since I never used C# or Unity before. I made some 3d renders of the characters and posed them to fit in my game. Using Python, I developed a machine learning algorithm that detects the image in the frame when you scan to determine if it is an insect, fish, or artwork. I used Google Cloud to help classify the images using reverse image search API and Google Art and Culture to web scrape information about the critters or artworks you were scanning. Lastly, I use DataStax to store the user information like what items they scanned.

Challenges we ran into

I had to learn how to use Unity for game development. It was probably the biggest challenge next to creating the AI that identified different species of fish and insects. The artworks were easier to use because there were more metrics including color, brightest/darkest point, form, style, etc. Insect and fish species are different but it is hard to develop an AI that can correctly identify the species of both.

Accomplishments that we're proud of

I'm proud to have made a game that drew inspiration from my favorite game, Animal Crossing. When I took my first break, I saw what I created and was excited about completing it the best I could.

What we learned

I learned how to use C# and Unity. I also learned how to use Google Cloud technologies that really helped with web scraping information about each item. Lastly, I learned how to make an AI that uses your phone camera to determine different insects, fish, and artworks.

What's next for Animal Crossing IRL

I hope to continue to work on this app. Currently, not every species of fish and insect can be scanned but I hope to crowed source this data to make the AI better at identifying images.

Built With

• adobe-xd
• ai
• c#
• datastax
• google-cloud
• machine-learning
• unity