Gen Art Recommendation System (GARS)

About the Project

Inspiration

We are a team of recent electrical engineering and computer science graduates, and our Generative Art Recommendation System (GARS) began as our senior capstone project. We aimed to push the boundaries of traditional recommendation systems, which typically suggest only existing content. With advancements in generative AI, we saw an opportunity to revolutionize this space by enabling recommendations for entirely new, never-before-seen content. We believe that our system can be used for tasks such as asset creation, making it a powerful tool for design platforms like Canva. In our submission, GARS Canva, we demonstrate how it can be effectively integrated with Canva using the Canva Connect API.

What It Does

The core functionality of GARS is to recommend images and artwork to users based on their preferences. Users rate each image on a scale from -1 to 1, with -1 being the worst and 1 being the best. These ratings provide feedback to our system, allowing it to generate art that better aligns with the user’s tastes.

Users can start the recommendation process with a random image, set their image preferences, or provide a starting point via a text prompt or image. After setting their preferences, they can customize options such as the number of recommended images and the granularity of the ratings. Ratings can be set to a simple scale of -1, 0, or 1, or a more granular scale that allows for finer adjustments between -1 and 1. Users rate images by swiping: swiping left or right adjusts the rating closer to -1 or 1, depending on the granularity setting.

To create a more complete design tool, we integrated an image editor that allows users to perform both standard and AI image editing on their recommended images within a canvas. The AI image editing functionalities include outpainting, inpainting, image-to-image transformations, and search-and-replace, while standard edits allow for adjustments to color properties. After editing, users can directly upload the modified image to their Canva asset library. If they wish to start over, they can restart the entire process. Additionally, if users find multiple images they like during a recommendation session, they can edit and upload each one to their asset library.

How We Built It

For our user interface, we used React and Shadcn for reusable components. The image editor utilized Konva, a Canvas-based library for image editing, and Stability Diffusion’s image editing API for AI-driven modifications. To upload images to Canva, we integrated the Canva Connect API.

To handle authentication and external API requests, we used a Node.js backend. OAuth 2.0 was implemented to securely authenticate users with Canva, ensuring that only authorized individuals can access and manage their Canva asset libraries directly from our application.

For our recommendation system, we cached the GARS session objects, enabling quick retrieval of the user’s current state in the recommendation process and ensuring seamless session continuity. Additionally, since users in a GARS session often move rapidly through recommended images, caching these objects allows us to efficiently retrieve the user’s current state and promptly return the next recommended image.

GARS also uses Milvus, an open-source vector database, to manage and search through the vast collection of high-dimensional vectors that represent artwork characteristics. As users interact with the system, their preferences are encoded and stored in a GARS session object, allowing GARS to quickly retrieve and recommend new art that aligns with their evolving tastes.

Finally, as our recommendation system suggests new art by adjusting the prompt-based representation of the artwork, we needed a text-to-image model to generate the corresponding artwork from the prompt. To achieve this, we chose to use Stable Diffusion's XL Lightning model hosted on Fal.ai, which offers rapid image generation, producing approximately one image every 2-3 seconds.

Challenges We Ran Into

One challenge that we encountered was the need to fine-tune our recommendation system’s parameters. We currently adjust these parameters by experimenting with different configurations and manually evaluating the results. In the future, we aim to incorporate A/B testing and use metrics like hit rate to systematically assess the effectiveness of our recommendation system across different parameter configurations.

Accomplishments That We're Proud Of

Our recommendation system is able to consistently converge on a specific topic or style of image by the end of a recommendation session. Achieving this level of accuracy in suggesting content that aligns well with the user’s preferences is one of the more challenging aspects of building a recommendation system. However, our system manages to do this within just a few iterations, making it both effective and enjoyable to use.

Despite not being UI/UX design experts, we are particularly proud of the user experience we've created. The entire app flow feels intuitive and logical, with features like the swipe-based image rating making interaction seamless. The image editor also exceeded our expectations, making it easy to perform basic and AI image edits.

What We Learned

We learned how to take an idea and transform it into a product that people can use. We never envisioned that we could take our recommendation system and turn it into a design tool. It was initially made as a proof-of-concept that such a recommendation system could be created and function well. However, after completing this hackathon, we believe that the potential for generative recommendation systems is vast, with applications far beyond what we initially imagined. These systems could revolutionize how users discover and create content, offering personalized and innovative solutions across various fields, from design to entertainment and beyond.

What's Next for Generative Art Recommendation System

The main application for GARS is still under development, but GARS Canva will likely become a feature within our broader design tool. We aim to transform GARS into a fully functioning application within the next year. Users can sign up for our waitlist on our demo website to be notified of our launch. If you have any questions/concerns/suggestions or would like to try a demo of our app, please email clevergars.info@gmail.com.