ArtStories

The app that tells you the stories behind art.

Inspiration

Every time someone stands in front of a famous painting, they ask the same question: why does this matter and what was the intention of the creator? At the same time, being live at an art museum, exhibit, and/or reading a museum tag never fully answers this question So, we wanted to build something that meets people where they are and answers those questions. You drop an image and get the real story behind it. With this in mind, we can find out more about why a piece of art was made, opening the door to you building a deeper connection with the piece and the artist.

What it does

ArtStories lets you drop any artwork image and instantly receive its history, cultural context, and meaning from an LLM and Wikipedia. It also generates an audio narration so you can listen to the story. Community members can publish their own blog posts about pieces they've created to share their stories, and when someone uploads a matching image later, ArtStories detects it and surfaces that story.

How we built it

We used modern free llm api's from google, runway, elevenlabs to generate the text, audio and video content, while using react vite and node to run the frontend environment with async functionality.

Challenges we ran into

The hardest problem was video. We originally tried to synchronize a video experience with the audio generated. We built toward it but couldn't get the timing to align reliably within the hackathon window. Realizing that it was too difficult to implement and wanting to focus on perfecting our other features, we made the call to create audio narration instead. On the API side, we discovered early that the Met Museum, our original data source API, doesn't hold most of the world's famous works. As a result, we changed our pipeline to go straight to the wikipedia API. One more thing we struggled with is trying to add a cropped image into our image generator. Gemini is great at detecting the image if the full thing is in frame, but if part of the image is there it can not detect it. As a result, due to to the high difficulty of making this happen, we decided to just mark it as no results being founded in order to prevent the AI from hallucinating.

Accomplishments that we're proud of

We're proud of being able to create an application that works within the few hours that we had here at Husky Hack and are proud of the skills we learned in collaboration with each other. Especially for pushing me (the designer) to learn more about coding and different tools. We are proud of our effort and collaboration.

What we learned

We learned that famous artworks aren't always where you expect them to be because an API that we thought would be more credible ended up being less reliable.

What's next for ArtStories

Expanding beyond famous works to support original and emerging artists — giving unknown pieces the same depth of story as a Van Gogh. Better video generation as the tooling matures. And a mobile experience so ArtStories works in an actual gallery, pointed at an actual painting, in real time.