66 _ Realtime interactive and emotional generative painting

Random generation of our trained model
Random generation of our trained model
Random generation of our trained model
Random generation of our trained model
Random generation of our trained model

Inspiration

Traditional art is usually static where the artist chooses the final product and sets the emotions and feelings during the creation phase and it stays like that forever.

Advancement in computer science and new media allows a new generation of artists to create artwork that's dynamic and interactive to connect with people's emotions and allow them to be part of the artwork itself in realtime.

We have trained an AI GAN model on ways to understand human emotions interactively through extracting facial expressions and generating an art piece that connects with the viewer's emotions and facial expressions in real-time.

What it does

Digital artwork on the wall connected to a camera where it takes a picture of the current viewer and extracts their fecial expressions and send that extracted features image to our trained GAN model to generate an art piece based on the viewer's facial expressions in real-time.

How I/We built it

1- Collecting Data:

We have collected high resolution carefully picked famous oil paintings from the period of 1400-1900, which include Renaissance, Impressionism, Symbolism, Expressionism art Movements with famous artists e.g. Leonardo da Vinci, Michelangelo, Raphael, and Donatello, Vincent Willem van Gogh, Oscar-Claude Monet, Pierre-Auguste Renoir, Gustave Moreau, Gustav Klimt, Mikalojus Konstantinas Čiurlionis, Jacek Malczewski, Odilon Redon, Pierre Puvis de Chavannes, Edvard Munch. We focused on artwork that expresses emotions and expressions.
Extracted the facial expressions/landmarks from all the paintings to be used for training the model later

2- Training AI Model: We have used Pix2pix image-to-image-translation to train our model on these collected paintings with their extracted features so the AI can understand human emotions from the selected training set/style.

Challenges I ran into

Processing Power: but luckily google cloud was there for the rescue.
Availability of high-resolution paintings: but lucky after extensive research we found good sources to get us started.

Accomplishments that I'm/we're proud of

That we have collected the data(Paintings) and trained the AI model ahead of time.
Working in a collaborative multidisciplinary team (Ai Artist & Data Scientist & Electronic engineer ) to create an engaging and interactive experience.

What we learned as

That our collective effort and skills across different domains (Ai/Art/Data/Engineering/programming) created completely dynamic and interactive digital artwork that communicates to human emotions in real-time by making them part of the creation process and ultimately the artwork.

What's next for 66 _ Interactive emotional generative painting

To collect more expressive paintings to improve our model to understand a variety of human emotions e.g. Happiness, excitement, worry, and sadness, etc...
Building the device: We have used Rassparry Pi connected to a camera where it takes an image of the viewer currently standing in front of the digital painting and extracts the viewer's facial landmarks using Dlib and send that to our model to generate an artwork based on the viewer facial expressions. link to the devise tool: https://github.com/obal3588/66_Artathon_RPI_IOT
Building the online demo tool: We have exported our trained AI model checkpoints to be used online using javascript to connect with user camera, take a picture and run it by Dlib to extract the facial expressions/landmarks to be sent to the exported pre-trained model and show the user the newly generated artwork from their facial expressions in real-time.
Use Pix2pix HD to train our model and generate high-resolution artwork.
Store and Generate the QR code of each artwork on the cloud so people can access and share their creations online.