AI-enabled Digital Heritage VR Game - Alqatt

Murals created with Alqatt in modern buildings in the Asir region.
The Alqatt Team at SAAI's Regional Hub in Riyadh - Working with Artists to Understand the Patterns in Alqatt
Alqatt artists were on-hand in the regional hub to co-design the machine learning models and to identify the relevant elements for training.

Inspiration

Inspired by the diverse approaches for applying AI in the context of art authentication, we are exploring the use of machine learning models for authentication of heritage art. In light of the 2006 sale of the inauthentic 1927 La Horde by Max Ernset at Christies, human connoisseurship of art has proven itself to be fallible, even when it's the best money can buy. This is why in cultural heritage and in preserving something as important as AlQatt, which is inscribed in UNESCO's list of Intangible Cultural Heritage of Humanity, is in need of the support of digital connoisseurship to verify something this meaningful and valuable. In our project, we have developed a proof of concept using the Google Teachable Machines and a dataset of images we have collected from activities related to preserving and teaching this art to the general population.

Google Teachable Machines is a tool that is web-based and was favored for this stage of the project not only for its speed, ease of use, and accessibility to develop this concept but for a pre-existing use case on European art authentication, particularly in identifying and classifying Vermeer paintings. This project was inspired by the filed work conducted by Dr. Haifa Alhababi in the Asir region in the scope of Alqatt artworks. We are taking the field work to the next level by digitizing the heritage, and developing gamified interactive products to make it more accessible to general audiences.

What it does

The VR game is designed with challenges in creating artworks aligned with the patterns that are recognized in the Asir region. The AI engine is designed to be embedded in the VR game to run the 'authenticate art' component for gamers to advance from one level to higher levels in the VR game.

How we built it

Unity Game Engine. Unity Machine Learning Agents Toolkit (ML-Agents) which is an open-source tool that enabled our VR game to serve as an environments for training intelligent agents that can recognize and authenticate Alqatt art in the interiors of the 3D environments that we built in Unity. We also used C# and Python for the scripts in the machine learning models.

Challenges we ran into

The main challenge we faced was collecting the data from the heritage resources for Alqatt art. Our team worked in collaboration with scholars at the Royal Institute of Traditional Arts via the Ministry of Culture in Saudi Arabia to help us source authentic Alqatt art visuals that we eventually used to train the machines.

Accomplishments that we're proud of

The first proof-of-concept of the VR game received the Ithra award https://www.ithra.com in the Creative Solutions Program competition in August 2021 https://www.ithra.com/en/special-programs/creative-solutions. In September, the team expanded with more developers to embed the AI component in the game design for the purpose of increased engagement and ensuring validity in the visuals that are used in the game.

What we learned

We learned how to segment images with the 'vocabulary' of Alqatt art, from the strokes to the semiotics of the symbols in Alqatt art through working with the regional artists (Alaa Maghawi, pictured in the photo shared on our project page, as she worked with the team during the SAAI regional hub event in Riyadh). From this foundation, the developers were able to create the dataset that was used to train the machine.

What's next for Alqatt XR

The game development team is working on the development of the VR game with Unity. The first release of the game is planned for November 2021. The Machine Learning team is working on the art authentication system to expand its training with more data as it becomes available from the Royal Institute of Traditional Arts as well as aligning it with the game's design via Unity's ML agents.