ArtLine – Harvard Art Museum AI

An AI that can voice call with you on anything about the Harvard Art Museum,from each artwork in the space to upcoming events as you plan your visit, applicable to customer service in any enterprise

Comment

Inspiration

  • Sardor is experienced in making generative AI chatbots for customer service.
  • Helen is passionate about art, museums, and literature and making them approachable.
  • Realization that large museums, as well as their websites, can be overwhelming for less experienced visitors.

What it does

An AI that can have natural conversations with you through voice calls on anything you train it with, our example being:

  • Your customized tour guide to the Harvard Art Museum that can:
    • Talk about any object in the collection.
    • Share any information about museum hours, admission, current exhibitions, events, etc. (that exist on the website).
    • Provide interesting, intelligent, natural responses with extremely short reaction time.
    • Give specific answers drawn from the website you train it with, instead of a generic one produced by LLMs.
  • Other potential applications being:
    • Online Student Support.
    • Company Customer service.
    • Plato (!) talking to you about different ideas that different characters hold on "love" in his work Symposium.

How we built it

3 Big Steps.

  • Draw from Deepgram to build an ASR application that you can voice call.
  • A. Scrape the website of the Harvard Art Museum through its shared API for its object collections and all other information ("Visit," "Event," etc.) and convert them into a vector space.
  • B. Retrieve the embeddings that are semantically similar to the question.
  • C. Use OpenAI's GPT to put together the retrieved information from the vector into natural human-like language.
  • Draw from Microsoft Azure to construct a Speech Generation Model that articulates the response to the user.

Challenges we ran into

  • The AI used to give a generic answer drawing from LLMs instead of information from the specific website.
  • Hardcoding the scraper of the website was time-consuming and inefficient, and fortunately, we find access to the API of the museum objects on the website.
  • How to shorten the reaction time caused by the turnaround time of the three AI models we use

Accomplishments that we're proud of

  • Extremely Short Reaction Time.
  • Human-like answer and relatively accurate response.
  • Bridging disciplines in the arts and tech.
  • Clear impact on improving user experience in any enterprises we apply the model to

What we learned

  • Deepened our understanding of the speech recognition model, speech generation model, and large language models by constructing.
  • Learned to deconstruct a problem into chunks and steps and combine complex language models into an application.
  • Experimented with endless creative opportunities at the intersections of different disciplines.

What's next for Artline

  • Contact Harvard Art Museum and solicit support in turning this program into a free accessible product (Helen plans to have an internship at the museum).
Share this project:

Updates