The problem our project solves

Virtual Tours are becoming increasingly important, especially so in this age of remote life imposed on us by the fight against COVID-19. Museums are locked down, their guides unable to work, many people are blocked at home, in need of intellectual and emotional stimuli. The goal of our project is to develop one application on mobile phones that will cater to the needs of cultural workers and audiences committed with cultural life, turning the current crisis into an opportunity both for Cultural Heritage institutions and for the people. Imagine a world in which Museums are energised by the crisis: quickly creating immersive digital content about their artefacts, allowing everyone on the planet to explore and enjoy their collections and to comment and build stories over the Cultural Heritage that most inspire them. This would foster digital social interaction and cohesion among people who are physically separated because of the current virus-induced situation.

The solution we bring to the table

The MEMEX project will create a mobile app with two working modalities, Virtual Tours and Augmented Reality Tours.

The Virtual Tours modality can be used both in general times and during periods of lockdown. It allows Cultural Heritage institutions to create virtual tours of their collections: Points of interest are created on the map of a museum, with each point containing rich content about the items that are physically on display in that area. Such content can include photos, 3D models, written descriptions as well as oral descriptions recorded by the guides of the museum. Sequences of artefacts are stringed together to form thematic stories, which assist the users in performing a coherent exploration of the collections. During periods of lockdown, people have a chance to perform such virtual visits to get to know the collections of local and global museums. Furthermore, visitors have a chance to tag artefacts with content they create. Such content can take the form of written text, audio or video, enabling the exchange of expression and information on how different people experience art and culture, and fostering new practices of social cohesion and cultural common imaginary. Creating virtual tours is attractive for Cultural Heritage institutions because it raises awareness of their collections and will increase the influx of visitors once the lockdown is over.

The Augmented Reality Tours modality builds on state-of-the-art research in Computer Vision and Artificial Intelligence to build an immersive story experience in urban contexts. People (because we aren’t just users) connect their memories to locations and Cultural Heritage artefacts by using a combination of GPS and advanced visual matching. As an example, a person can use our app to tag a statue in the centre of their city, attaching original content to it. The content can be in the form of written text, audio or video, and relates to the personal significance the statue has for the user. When other people use the app in the same location, they can create new content, as well as visualise content previously contributed by other users, in the form of Augmented Reality. Enabling the exchange of information on how different people live the areas of a city will foster social integration. This is particularly important for people belonging to vulnerable social groups, which have the highest probability of being disconnected from the society during times of social distancing.

What we have done during the weekend

In these two days we refined and reformulated our vision, driven by the goal of making our app as useful as possible in the time of COVID-19. We discussed how to best exploit the synergy between the two modalities of out app to that end: the Virtual Tours modality, especially useful to entertain people and to publicise museums during lockdown, and the Augmented Reality Tours, which offers a chance to people in socially vulnerable groups to form new social relationships thanks to digital interaction, once the lockdown is over. We took inspiration from the talks offered by experts, from browsing the proposals of other projects, and from the interaction we had with some of the fellow hackers. Discussing with our mentor helped us focus on the business aspects of our project. We brainstormed possible revenue streams, including their ethical impact (Is it acceptable to embed ads in our app, or to profile users? Would it be possible to keep the app free of charge for the smaller institutions?). We defined strategies to generate adoption of our app by museums, evaluated the effort required on the institution side for publishing content, and the fee that would be required for making the app financially viable. We discussed the effort that would be needed for scaling the distribution of the app, i.e., the level of effort that would be required for extending the service to a new city, or to a new country. We defined steps to make our estimates more concrete, such as getting in touch with Cultural Heritage institutions and assessing the level of their interest in our app, the time frames they would need to produce digital content, their willingness to pay a fee for the service our app will provide.

The solution’s impact to the crisis

The app has the aim to improve social cohesion and to foster cultural activities in cases of further lockdowns. It represents a chance for cultural institutions to open up their collections and to attract virtually any person that cannot, for any reason, physically access our invaluable cultural heritage.

The necessities in order to continue the project

We are using Mapillary as the backbone ( to build our app. The users will be able to see their cultural heritage assets of interest on the app. In the beginning, we will use Mapillary's API as they are very stable and tested on the ground. The app will be built in two further stages:

  • First version: it will allow visualising cultural heritage assets located in the vicinity of the user. The user will also have the chance to select the artefacts to be visualised by browsing the map. Museums would be able to provide their content, while some of the content would be automatically fetched from the web and geolocalised.

  • Second version: it will allow users to create geolocalised stories and narratives over objects that can be visualised with Augmented Reality or simply browsed on the map.

As we are funded by an European project for the basic infrastructure deployment, we will need extra funding initiatives in order to create a stable system connecting millions of people with our app.

The value of your solution(s) after the crisis

We believe that the impact of our app on society after the crisis will be threefold. First, we expect that the Virtual Tours that will take place during the lockdown will contribute to stir the interest of people for visiting museums in person. People will take advantage of the extra free time during lockdown to get to know the Cultural Heritage institutions in their areas, and will visit the ones that interest them the most in person, when they will be allowed to do so.

Second, once people will have more freedom to move and cultural institutions will reopen, the Augmented Reality Tours modality will be deployed. That modality will allow people to interact with artefacts located in the outdoor areas of the cities, by creating and consuming content and associating it with statues, monuments, and points of public interest. The integration of the two modalities in one single app will help attract people to the museums: while people will navigate outdoor areas, they will receive information on user-provided content both in public places and inside nearby Cultural Heritage institutions.

The third effect of our app after the crisis is related to the (re)-integration of people from vulnerable groups into the society at large. The app will provide audiences that are socially fragile and systematically blocked from various cultural opportunities and resources with a tool to express themselves. It will allow them to be heard. The app will help building a whole new digital community, centered around people’s experiences and memories, and the Cultural Heritage that inspire them the most.

Built With

+ 28 more
Share this project: