The main reason we developed this project is because painting is a very difficult skill and there are a lot of people who are passionate about painting but not so good at it. So we wanted to see if AI can be helpful in delivering the art of painting to a larger audience. This platform can be a very good start for those who wish to paint but are not too skilled at painting. More than all of that, it's always fun to allow a computer to finish our painting!
What it does
- pAInt packs an MS Paint like canvas with which the user paints a rough sketch.
- It then converts it into a realistic version with different styles. New styles can be obtained on refreshing the screen every time.
- It can add an artistic touch to the realistically synthesized image.
- The artist can also upload a design or another picture from which styles get transferred to the artist's original painting.
- There is also a complete gallery/cart experience that is built into the application
How we built it
Pix2Pix model was used for generating a realistic image. Cycle GAN was used for producing variations of synthesized image, Bicycle GAN for the artistic touches and Neural Style Transfer for customized artistic touch. All models were built using PyTorch alone.
The application was built using TypeScript, that calls the python script that uses Pytorch with the models that we built. The frontend, built with VueJs communicates with the REST endpoint exposed by the TypeScript app to complete the user experience. We have used AWS's EC2 for hosting the web application
Challenges we ran into
- Training with GAN as a whole including mode collapse and stability/convergence issues while training it.
- The lack of styling according to personalized input in the synthesizing model posed a problem which was overcome by the introduction of Cycle GAN.
- Devising the segmentation maps for different classes and mapping it between different datasets.
- Forking jspaint and customizing it to suit our needs was the most challenging part
- Setting up the AWS server with all the dependencies; setting up the GPU.
- Linking the backend with the front end and all the communications was a bit challenging
Accomplishments that we're proud of
- Building a stand-alone end2end application which can help lots of creative minds (and idle minds too xD).
- The very thought that we have our application that is accessible over the internet is a very cherishing one
What we learned
Strong theoretical and practical implications of GANs, Working with AWS, NodeJS, VueJS, Integrating multiple models together, building an API to suit the needs and the usage of pytorch in real world ML problems...
What's next for pAInt
- Enable synchronous painting access that enables two artists to complete an art in the canvas over the internet
- Checkout being made as paid which can support local artists.
- Delivering printed art to the customer... (We are being really ambitious here! xD)