My AI Artist-Collaborator

Can I collaborate with an AI, and not just use it as a tool? Can the AI learn from me and generate new works? Can I use the AI's works for inspiration to create new ones?

Comment

Inspiration

AI Generated art has exploded in numbers recently, especially with the tremendous advances in GAN architectures. Many artists have used AI to produce weird, wonderful, unexpected, and unsettling images. Many other artists have tried to deconstruct how AI systems produce and understand images, and what their biases are. However, true collaboration between human artists and AI artists as equal partners remains underexplored. "My AI Artistic Collaborator" proposes one such model.

This partnership creates a continuous loop between myself and a custom AI artist. The AI learns from a body of work I produced (my "PCB Drawings" series); it then produces new works inspired by mine; and I respond to the AI's works by producing new works. In the end, all these new works go back into re-training the AI and restarting the process, thus making our styles and artworks move ever closer to each other's.

What it does

The result of the hackathon is a working prototype of an app that can generate new circuit board artworks. Over the past five years, I have been developing new artistic methods that produce non-functional circuit boards that are designed purely as art objects. For the hackathon, I have written an app that uses a neural-network model trained on all my existing circuit board artworks. The app can either generate a circuit board from scratch, or, if the user has started a circuit board, can suggest completions for it.

As part of the hackathon, the AI model has generated some boards, and I have used these purely-AI artworks as inspiration to create new works. Thus, this hackathon implemented most of the first pass of the proposed model of human-AI collaboration. I want to manufacture some of both the AI-created and AI-inspired, human-created works.

In addition, the web app developed for this hackathon can be deployed publicly, and the public can use it to generate their own circuit board artworks. Using the app like this also raises some questions about the authorship of these circuit boards: am I the author (since the training dataset consists only of my artworks), or is the AI the author (having actually generated the circuit boards), or is the user the author (having had to sift through some generated models until they found one they like)? All three of us?

How we built it

There were three distinct phases for this hackathon.

First, I encoded the computer models of all my circuit board artworks (all developed in Eagle CAD) into shorter strings to create a dataset of all my artworks. This dataset was then trained with a slightly-modified Textgenrnn network to create a model file. Using this saved model, new circuit boards could be generated. For this part, I used Python and various XML libraries; I also wrote and/or used several character-level text generators (all in Tensorflow) to train on the dataset. I picked character-level text generators because the Eagle XML models are heavy on numbers, and it would be really hard to build a vocabulary out of all possible digit combinations.

Second, I created a web app for circuit board generation. The app (which is not yet deployed publicly) can either generate new boards or suggest completions for partial board models using a Textgenrnn model. To implement the app, I used Flask for the server back end, my own libraries to handle Eagle XML transformations, a partially finished Python library that transforms Eagle models into SVG files, and Imagemagick to render these SVG files into PNG images.

Finally, once the web app worked, I have started to generate some circuit boards and create new circuit boards inspired by the AI-generated ones.

Challenges we ran into

The biggest challenge was finding a good NLP network architecture that suited this project. I have tried 3 different architectures:

  1. A simple 2-layer LSTM + 1-layer dense architecture.
  2. A mini-GPT model, slightly modified for this project
  3. An existing character-level generator (Textgenrnn), again slightly modified and adapted to my project

Of these 3, the Textgenrnn network worked the best (the demo uses this model). The simple model was hardly able to produce correct circuit boards (the XML was incorrectly formatted). The mini-GPT model was the biggest disappointment: I really believed that such a model would be able to learn and capture effectively the circuit board models; however, the quality of the boards was again extremely poor. It is possible that I have too few circuit board examples in my dataset and the model has too many trainable parameters to learn well from such a small dataset.

Accomplishments that we're proud of

I believe that this hackathon project proved that my proposed model of collaboration with an AI is feasible. I also believe that this model could be a source of inspiration to other artists, who can adapt the general framework to their own style of making art (for example, could a weaver use their own patterns to train their own custom AI and collaborate with it to create new ones?).

What we learned

  1. I've learned the basics of natural language processing
  2. I wrote my first neural networks from scratch, as well as modifying existing networks and packages to suit my needs.
  3. I am much more comfortable using Python now, having written a fairly substantial code base for this project. I've also learned how to build sites with a Python back-end.
  4. Above all, I've learned that this idea for artistic collaborating with an AI artist is not far fetched. Yes, it's still early, and the AI's works can be refined further, but my biggest takeaway is that this project has legs.

What's next for My AI Artist-Collaborator

  1. I would like to create/use a better language network for my models (there's still a large percentage of boards that are not correct). For example, I want to try a full implementation of a GPT model (see here). I would also like to collaborate with someone much more versed in NLP -- I believe this collaboration would help me avoid many pitfalls and dead-end paths.
  2. I would like to deploy my site publicly and offer anyone the chance to try and generate circuit boards as art objects.
  3. And, above all, I would like to manufacture and frame these boards produced by the AI and have a two-artist exhibition IRL with their and my boards -- hopefully, by then it would be hard to distinguish who created which artwork.
Share this project:

Updates