Artisian

Before: "a happy snowman in front of mountains"
AFTER: "a happy snowman in front of mountains"
BEFORE: "a futuristic gaming chair in front of winter mountains"
AFTER: "a futuristic gaming chair in front of winter mountains"

Inspiration

Do you ever have an idea but don’t know how to visualize it or feel restricted by your lack of artistic ability? Artisian is here to make creativity accessible to everyone. With Artisian, simply draw a rough sketch of your idea (you can also add an optional text prompt), and we'll automatically generate an enhanced version of your sketch, just as if it were drawn by a professional artist! Artisian is extremely straightforward to use so anyone, even a 7-year-old kid, can become a professional artist with a few brush swipes and the click of a button in less than 60 seconds.

Currently, there are a few diffusion models available online, but all of them are very limiting (purely text-to-image or image-to-image). None of them provide you with an option to draw on an empty canvas in the way that Artisian does and help you bring a sketch to life in less than a minute.

What it does

Our application adapts a simple and easy-to-use user interface. You start off on a blank canvas, on which you can draw using our selection of painting tools and colors. After you are done drawing, you can add additional information by typing a description of your drawing in the text box below. Then, click the generate button. This will take a capture of the drawn canvas and feed it into our diffusion model, which will process and generate a refined and high-quality image of your original sketch.

How we built it

We first wireframed/prototyped our design using Typedream and Figma. Then, the front end was coded using HTML, CSS, and Javascript. The backend is based on a Stable Diffusion model that does image-to-image translation pretrained on billions of data samples, using PyTorch in Python. We then used Flask in Python to connect the frontend to the backend.

Challenges we ran into

None of us have previously worked with diffusion models. We also didn't really have much experience doing web development, so it was a big learning experience just trying to figure out how everything works. The diffusion models are also very large, so it took a very long time to download them and run them locally on our computers.

Accomplishments that we're proud of

We are super happy to have built an end-to-end website leveraging the power of these models in a clean web interface. It is so fun to play around with the images generated, and we truly felt like artists ourselves! We really believe that this website has the potential to positively impact many individuals, and help create a more welcoming place for those that do not consider themselves so creative or artistically talented.

What we learned

One super important lesson we learned is that less is more. We broke down to the simplest features possible essential for the user, so that we could come up with a great user interface. While we had the option to display more advanced features on our apps, such as fine-tuning the weights and guidance scales, these terms were too technical for the users, so we decided not to overwhelm them with the final product.

What's next for Artisian

We would like to deploy this technology to a mass audience on a website. We also would like to add options for customization (in a way that is intuitive but not overwhelming), as well as generate multiple images simultaneously. This is something we can do on the backend, but we did not manage to finish implementing that in time for the frontend.