When visiting museums, sometimes we found that it could be hard for people, especially those non-native speakers, to understand the annotation and the underlying meaning of the artwork. Thus, we came into the idea to build a mobile app that provides people with quick and succinct translation help in museums.

What it does

The app will first lead to a language selection page for the user. Then the user can choose either to scan the artwork or scan the annotation of the artwork (usually next to the artwork). Then the user will get the title, brief introduction and a button to click on if they would like to learn about the artwork. They will also be able to trace their visit history, which will generate a list of artworks the user has searched.

How we built it

We used React Native for frontend and Node.js for backend. The frontend receives user's input such as iamges of the artwork and preferred language, and sends them to the backend. At backend, the app uses Google Cloud Vision API to identify the artwork in the image user uploaded, or extract text from the image by OCR. We then search these keywords using Wikipedia's API and render them on the screen.

Challenges we ran into

We were having trouble in handling different types of feedbacks given by Google Vision API (sometimes they could be as specific as the name of the artwork, but sometimes they are garbage information ). We solve it through adding additional api calls via wikipedia internal search in order to get a more precise and concrete result.

Accomplishments that we're proud of

MULtiSEUM greatly reduces the valuable time user spends on typing the words in the search engine to get the necessary information when visiting the library. We are proud that our application helps a variety range of population by providing localization and accessibility functionalities.

What we learned

We taught ourselves a bunch of new skills and technologies that we end up being really interested in.

What's next for MUiltiSEUM

We hope to introduce more functionalities, including location track and more detailed classification of artwork.

Built With

Share this project: