VinGo - interactive museum excursions

• 

  Main page

• 

  Museum page with achievement and export progress

• 

  Recognizer interface

• 

  Recognized image

About APP

😷 Pandemic changed the world and we need new approaches to ordinary things. ⚡️ We reinvent the museum excursions because peoples want to see world art culture but make it more interesting and safe 👮‍♂️.

🔥 Vingo is application that will your own guide in museums. Vingo gives u a emoji-puzzle, that's mean some picture that contains this emojis in the room and u need to find that.

For example 💃💃💃💃💃 - is Matisse's picture "Dance". We see five dancing persons in the picture and in the emojis.

Also 🚣🌊 ⛅ - Ivan Aivazovsky, "The Ninth Wave".

👀 After u find the picture, u need to put ur camera to the picture and that will recognize, and u see a short Instagram-

😉 This application allows making interesting museum excursions without big groups of peoples.

Challanges

Our applications have got a realtime picture recognizer. The main challenge is to make zero-shot picture recognizer in realtime on mobile. Because we have got only one image per picture (we scrape all images from the Hermitage museum site). Also, we want to recognize pictures on device from video, without taping on any buttons and sending data on the server.

How its works

To train recognizer we train mobilenet by PyTorch lightning on WikiArt dataset to predict the genre of this picture and then we tune the model on metric learning task, to make good embeddings for pictures.

To infer network on the device we use torch-jit for IOS. We get a cropped image from the camera every 0.5-sec end send it to the network to get embeddings vectors. And then just find the nearest image by KNN from the database. To reject false-positive activations we use a strong distance threshold.

Built With

• pytorch
• swift