Try it Yourself
Check it out yourself at https://fieldtripar.web.app/. Note that this requires iOS 11 or later to use.
Sometimes the best way to understand something is to experience it firsthand. That’s why museums offer such a great environment for learning and discovery. Unfortunately, COVID-19 has closed the museums that create and offer those valuable experiences.
Although some museums offer virtual tours of some of their exhibits, the vast majority are still only available in-person. A tiny number of museums enhance a few of their exhibits with augmented reality, but it can only be used inside the museum when you are standing next to the actual artifact. To date, there are no augmented reality tours of world museums that can be accessed from home. AR Field Trips will allow museums from anywhere in the world to spotlight items in their collections for a worldwide audience.
Our goal was to create a way for curious people everywhere to have those (sometimes once-in-a-lifetime) museum experiences by allowing them to explore artifacts from museums all around the world from their homes in a way that was as close as possible to actually being there by using augmented reality.
What it does
AR Field Trips allows children all around the world to experience museums’ collections from the safety of their own homes. It combines description plaques, audio tours, and AR experiences to create a realistic and fun environment for exploring and learning about a museum’s virtual exhibits. Museum curators can select a 3D model, an audio tour, and the informational text for an exhibit, and then the app generates a unified experience for the user. This system will allow museums to create virtual exhibits that are just as exciting and interesting as the in-person ones.
How we built it
To maximize the accessibility of AR Field Trips, we wanted to build a web app that allowed users to access the content that we provide without needing to download specialized software, or to purchase specialized hardware. In addition, we know that most of our users will likely be accessing AR Field Trips via a mobile device, so responsiveness is a key design feature. Therefore, we chose to build a React app, styled in Material UI so the website is accessible and easy to use on all sized devices. It’s also hosted on Google Firebase Hosting, to guarantee high availability and to automatically scale server resources up and down based on demand.
For the AR Display, we use Apple’s ARKit API, because it A) provides a robust web-supported SDK, and B) because we don’t have an Android phone that can run Google’s WebXR API. In the future, we plan to implement both for cross-platform compatibility.
Challenges we ran into
This was our first time using AR, so one of the hardest parts for us was determining which platform to use. We knew that we wanted the end product to be a web app so that anyone could access and use AR Field Trips without needing to download any specialized software. Consequently, we spent a lot of time playing around with different AR solutions to find the one that worked best and was the most accessible.
Most of the solutions we investigated required that the user place the AR object on a “mat” located in the real world. The object was scaled to the size of that mat, and went away if the mat left the field of view of the camera. As a result, most of the models rendered the same size as the letter paper that we printed out to act as the “mat”, and it just wasn’t very exciting to look at a 22 foot long dinosaur that displayed the same size as a common toy dinosaur.
We finally decided on the Apple AR Kit, because it offers us a high-quality AR experience, even if we need to then later create an Android experience. We wanted to make sure we didn’t compromise the quality of the experience. This meant that we needed an AR platform powerful enough to automatically place the image at scale to maximize the realism, and intelligent enough to keep track of where the model was even if the “anchor point” where the model touched the ground used to be was no longer in the camera frame.The end result was totally worth it, and the increased realism made the experience infinitely more enjoyable, even if it wasn’t quite as accessible.
Accomplishments that we're proud of
For a large chunk of the hackathon, we were really concerned that we wouldn’t be able to overcome the technical limitations many of our early AR platforms shackled us to. Therefore, we’re really proud of the fact we got everything to render at real scale, and you can interact with the model (like walking inside of it).
What's next for AR Field Trips
Now that we have the proof of concept complete, our next main objective is working on features to scale the app up. This includes upgrading the dropdown menu currently used to select which exhibit to view into a robust search system that allows users to filter exhibits by the name, the topic, the museum the real exhibit is housed in, whether or not there is an audio tour, etc.
We’d also like to build out a robust portal for museums and other content curators to securely upload and manage their own models. This will allow AR Field Trips to rapidly scale the content it can provide to users without requiring manual dev work to add each new exhibit to the website.
Finally, in the future we want to look again at AR platforms, to see if another platform either comes out or improves enough that we can increase our accessibility without compromising the quality of the experience that we provide.
The social effects of the COVID-19 pandemic will be with us for the foreseeable future. Even after the lockdowns imposed to contain the spread of the virus are lifted, many social distancing restrictions will still be in place. Attendance at public places like museums will be severely limited.
Museum-lovers everywhere will be able to use AR Field Trips to bring the museum artifacts to them, whenever, and wherever.