Milton

An app that makes art interactive, where you can get context/education information from artists by scanning. Making galleries more accessible with text-to-speech. Powered by Landing AI's Landing Lens.

Comment

Inspiration

Named after 17th century poet John Milton, Milton is an app that helps you learn about art. Much of Miltons's work was written completely in the dark, as he lost his sight in his 40s. Yet in his blindness, his work grew more vivid and imaginative. His magnum opus, Paradise Lost, is a testament to the power of art to transcend the physical world.

What it does

Milton is designed to help you experience art in a new way, by providing audio descriptions of art pieces. All you need to do is scan a piece of art, and Milton will provide you with information about the art and an audio description if desired.

How we built it

Stack: React Native, AWS RDS, Django, and Landing Lens by LandingAI.

Steps

  1. Collect data on art pieces by taking videos then extracting frames with ffmpeg.
  2. Train our model with Landing Lens. This involves drawing bounding boxes, dividing the images into a train, dev, and test buckets, tuning the model, analyzing where loss was occuring, and making economic choices as we spend our 1000 credits in Landing Lens.
  3. Build infastructure to facilitate a RESTful API for processing image data and fetching information of an art pieces from our database.
  4. Create a user friendly user interface.

Challenges we ran into

The biggest challenge is maintaining prediction accuracy while increasing the number of art pieces the model can detect. Transferring image data over https is also relatively slow compared to doing all processing locally on device.

Accomplishments that we're proud of

We were able to detect serveral art pieces with the model we trained. We also built a solid foundation for the project to be further developed. We know object detection models like the one used in Landing Lens is capable of precisely identifying art pieces as shown by a Google AI Research Project conducted in 2021. We hope to bring this technology on campus in galleries such as the Paul W. Zuccaire and Wang Center.

What we learned

We learned how to build web applications with Django, a framework that differs greatly in architecture compared to express.js. We also learned how the data used to train a machine learning model can have both positive and negative impacts on prediction accuracy.

What's next for Milton

Milton will be coming to a gallery near you! As Milton develops we will be moving to on device models for faster predictions and reduce latency in our UI.

Share this project:

Updates