Musezen - AI Art Curator Powered by Snowflake Arctic

Musezen Demo App

Inspiration

When visiting museums (or "musezens" 👀), have you ever wondered what more you could learn about a painting that only has "Oil on Canvas" written on its plaque? We have! Often, we end up looking up information on our phones, which feels like an interrupted experience. Musezen aims to change that.

What it does

Musezen is your personal art curator, designed to enhance your museum visits and art discovery experiences. Leveraging the Snowflake Arctic model, Musezen allows you to chat about various art styles and gain deeper insights into artworks.

How we built it

We built Musezen using Python with Snowflake Arctic as its core language model and Streamlit as UI. Our goal was to keep the project lightweight, avoiding unnecessary packages, which required developing some infrastructure ourselves. We designed an expandable class structure to support integration with different LLMs and to enable future tool use with a fine-tuned Arctic model. For image classification, we used ResNet-50 1 fine-tuned on an art style dataset 2.

Challenges we ran into

Developing Musezen involved both an image classification component and a generative model component, requiring parallel workstreams due to the tight timeframe. Each presented its own challenges. For the image classification model, we had to determine an appropriate level of label granularity—detailed enough to be useful yet feasible to train to adequate accuracy within the deadline. On the LLM side, we faced the challenge of integrating the classification model's output to provide as much insight as possible in response to user queries.

Accomplishments that we're proud of

We are proud of these implemented features with limited time (and thanks to Streamlit community cloud, the app can be used on mobile devices in actual museums):

Chat About Art: Start a conversation with Musezen about different art styles, artists, and historical contexts. Whether you're curious about Impressionism, Baroque, or modern art movements, Musezen has you covered.
Photo Uploads: Upload a photo of a painting to get specific information about the artwork. Musezen can identify the art style, explain its significance, and provide more information to spark in-depth conversations.
Interactive Learning: Use Musezen to explore art topics in a more engaging and interactive way, making your museum visits more informative and enjoyable.

What we learned

Tool use is a key ability for language models to enhance user interactions and provide comprehensive insights. Working on Musezen highlighted the importance of modular design and the value of integrating various technologies to create a cohesive and efficient system.

What's next for Musezen - AI Art Curator Powered by Snowflake Arctic

We are excited about the future of Musezen and have several enhancements planned to make it even more powerful and useful:

Art Database Connectivity: Leveraging high-quality data will unlock the full potential of Arctic, making Snowflake an ideal platform for such applications with close access to both data and models. We plan to start by integrating extensive art databases such as WikiArt, the National Gallery of Art Open Data Program, and The Art Genome Project by Artsy. Eventually, we aim to implement an easy way to allow smaller venues to set up their own versions of Musezen using their own data. This will make Musezen more useful and context-aware, enriching the user experience no matter where they are.
Enhanced User Experience: We plan to improve the user interface to make it more intuitive and user-friendly, ensuring it is easy to navigate and find the information needed. We also hope to support audio inputs and outputs to reduce the time spent looking at the device, allowing an even smoother experience in the venues. Finally, we aim to continually improve our image classification model to provide more detailed information about the user-presented artwork.

Authors

@AOYW: Zihan (Andy) is a data scientist with a B.A. in Economics and Data Science and a B.S. in Chemical Biology from UC Berkeley, along with an M.A. in Economics from Duke University. During his academic tenure, Andy contributed to several university research groups. Beyond working and observing his two cats, Andy enjoys exploring diverse cuisines and capturing experiences through photography.
@mittyhainan: Hainan is a data scientist with an S.M. from Harvard and a B.S. from UCSD. He has experience in diverse projects across various fields, ranging from e-commerce and healthcare to credit risk and business banking. Outside the office, Hainan enjoys skiing during the snow season, traveling to explore local cultures, hiking, and attending music festivals, especially EDM.

We hope you enjoy using Musezen as much as we enjoyed creating it. Happy exploring!