In video games, the 3D shadow puzzle can sometimes be found and is always a pleasure. However, we've yearned for more interactivity, which is why we chose to bring this puzzle into augmented reality.
What it does
Shadow Realm is an augmented reality mobile game. The app features a list of puzzles, organized in easy, medium and hard levels. Each level has a corresponding physical object, and a silhouette which is the target. To complete the puzzle, the user must rotate the object and cast its shadow onto a white surface to mach the target silhouette. Then they take a picture of the shadow and the app will confirm if the silhouette matches. If the shadow is similar enough, then they pass and move onto the next puzzle. Otherwise, the app shows by percentage how similar the shadow was to the target, and the user can try again.
How we built it
We used Java to build a native Android app. To detect similarity between images, we used openCV. To create our models we used modelling clay and imagination. Our display stand was made out of paper and HTN swag! ????
Challenges we ran into
Finding a suitable image comparison algorithm was difficult. Our first attempt involved using a perceptual hash, a hash of images that represents how it looks. However, we used the java.awt.Image library, which does not exist in the Android Sdk, so we had to let go of much of our work. Then, we tried to implement a perceptual hash using an Android graphics library, but we found that it was slow, and was too strict. During this time, we also had to implement functions to reduce images to black and white pixels, and constrain their size to fixed dimensions for better image comparison results. Alas, this was insufficient for our needs. Thus, at 3:00am today we decided to use openCV and used a Simple Eucliean Distance algorithm to image comparison. This option was must faster, and yielded far more positive results. Additionally, half our team had never worked on Android before, and the other half had worked on Android once before. We went to both the Android 1 and Android 2 workshops to learn the basics.
Accomplishments that we're proud of
We're proud that we were able to fulfill our vision despite having very little Android experience. We were also able to research several different ways to compare images, and were able to find a solution we're satisfied with.
What we learned
- How to make Android apps
- image comparison is hard and complicated
What's next for Shadow Realm
- New puzzles that involve hand shadows, multiple objects, and even people
- Hard puzzles where no target image is provided, and the shape needs to be figured out.
- Building a proper stage with a direct light source and platform to put objects
- user submitted puzzles
- selling the app to escape rooms, city scavenger hunts, and other augmented reality games
- improved image recognition to handle more image transformations. Possibly use maching learning for more reliable results.