We created AR World to make AR more accessible for everyone. Through the mobile app, any static image in the real world is instantly replaced by a video. Through the web app, users can upload any image and video pair to be replaced in real-time. The end result is a world where movie posters turn into trailers, textbook diagrams become tutorial videos, and newspapers/paintings come to life straight out of Harry Potter. These are just a few of the endless possibilities in how AR World can improve and transform the way we learn, entertain, and share information.
What it Does
AR World is an Android app that recognizes images seen through the phone's camera and seamlessly replaces images with its corresponding video using AR Core. It also includes a React web app that allows users to upload their own image and video pairs. This allows businesses like publishers, news companies, or museums to create content for its customers. Individual users may also upload their own images and videos to customize their experience with the app.
How We Built It
We built the mobile app using Android Studio, with Sceneform and AR Core on the backend to recognize images and map them to the corresponding videos. In the first pass, we recognized certain static images and replaced them with the appropriate videos. Then, we built a web page and API to accept more photo-video pairs that can be identified by the cameras of users. Thus, we needed AWS S3 to store these photos and videos, MediaConvert to convert them from MP4 format to a streamable DASH-ISO format, and CloudFront to serve the video streaming requests. Also, MongoDB was required to store a map from the image to the corresponding video link on S3.
Challenges We Ran Into
There were several challenging aspects to our project. To start, streaming resources from AWS, GCP or some cloud storage provider onto a device on demand proved to be a hurdle.
- Getting the video to stay anchored in the real-world
- Recognizing a static image
- Automating the conversion from user-uploaded mp4 to a streamable format
- Dynamically updating the image-video pairing database
Accomplishments That We're Proud Of
I think it's an accomplishment to have a fully functional app that can effectively recognize images within a certain set and replace it with the desired video from end to end (where the back-end, front-end, and infrastructure is complete).
What We Learned
I think we learned that there are a lot of unexpected issues when connecting different resources, and it takes a lot of patience and debugging to work through them. None of us had worked with Kotlin or AR before!
What's Next for AR World
For AR World, the next big step would be to create organizations in which people can be enrolled. This would help assign a group of picture-video pairs that pertain to a group (e.g. a group on a museum tour, to see more information about artifacts in the museum).
Another big step would be to give the full VR experience to the customer. With headsets like Google Cardboard, we can help people explore different parts of their environments simultaneously.