Art Echoes

• 

  Art Echoes Banner

Inspiration

It's common these days for students to listen to music while working - whether it's writing, programming, drawing, or exercising. Sometimes, it can be hard to find new music to listen to, and it can get boring if artists from your favorite genre are following similar composition trends. Art Echoes uses a unique way to generate original music - by using brushstrokes!

What it does

Art Echoes is a plugin for the drawing software Krita that converts brush strokes into musical compositions, creating a unique auditory experience based on your artwork. For more information on how to install and use the plugin, watch our demo video or read the README here!

How we built it

We used the Krita Python module to get Krita information like the currently active window and canvas; the rest of the brush stroke stuff was done with PyQt - we set up a small dummy widget so we could access (we hesitate to say "overload") the Qt event hooks. We used a hook to follow the user's cursor movements and determine when they had their left mouse button down inside of the canvas; closing the dummy widget stopped the collection of coordinates from mouse-down'd cursor movements and sent them to the music generation side of things. We used midiutil and mingus for creating the music, and then some math and music theory to turn coordinates into notes.

Challenges we ran into

Let's just say that the experience of trying to get coordinates on brush stroke events was five times harder than OSU's OS 1 class. Krita has their own main documentation called the LibKis library for Krita-related things, like layers, brushes, and the Krita canvas. This is written for C++, but it's relatively straightforward to convert into Python, once you get the hang of it. Then there's the Qt documentation - Qt is the underlying graphical stuff that Krita is built on. For the majority of plugin purposes, this is probably also not too bad - there is Python documentation for PyQt, though some of it is auto-translated from C++ as well.

The problem was the specific functionality we were after - getting information about brushstrokes. Going into the project, we thought the Krita API would have direct resource representations for a brushstroke and built-in hooks to do things on stroke. In actuality, no such thing existed - we had to spend hours digging down into the trenches of Qt's events, objects, and listeners. Because Krita plugin development is relatively niche, our resources were somewhat limited - we weren't able to find tutorials directly targeting what we were looking for, and the [krita-artists.org] forum had to take the place of Stack Overflow. On top of that, Krita doesn't have its own pip package, so we had to leave our IDEs full of squiggly lines until the next re-run of the plugin - which in itself requires re-copying the plugin files out to the resource folder and restarting Krita. The simplest way to import external libraries from pip that weren't built-in to Krita was to copy the whole package directly into the resource folder. One of our team members also unfortunately got sick (get well soon!), so we lost part of our humanpower.

Accomplishments that we're proud of

We were very excited about this project, and despite sacrificing more sleep than we would have liked, we did make something we are proud of. It was the first time for all of us working on a Krita plugin, and we went in as complete beginners to the world of Qt. We were also happy to have found a good team and to have stuck through all the difficulties together, despite living in different parts of the world. We're glad that we could accomplish what we did and are excited to see what else we can do with Art Echoes in the future!

What we learned

• There are more things than you'd think that sound like they would be easy but then take hours and hours of your time.
• Then there are also more things than you'd think that look very hard and confusing, but aren't that bad once you get into them.
• Good documentation can be very, very important :)

What's next for Art Echoes

• User experience improvements!!
  • We had to make an empty widget to attach the event listener for brush strokes, but it looks pretty ugly (and it appears whether the script is run or not).
  • The script currently creates a new 1000 x 1000 canvas for the user to paint on - the ability to run the script on any canvas would definitely be nice.
  • Right now the generated midi file has a hard-coded name and gets output to the Krita resource folder, which is not a place most users want to know about. It would be nice to let the user choose their own name and location for the output file.
  • A menu to control settings would also be nice. It would also be helpful to let the user stop and start new sessions of stroke-to-sound.
• More options - currently, brush coordinates are the only thing factored into the resulting note calculations. Some possibilities for additional properties to factor in are:
  • Brush -> instrument
  • Stroke speed -> tempo
  • Brush size -> volume
  • Color -> tone elements (e.g. sustain, reverb, etc.)
• AI-generated music?
  • In the future, we might want to look into using an AI model (e.g., Magenta) to generate the music instead of using formulas to generate the notes - this way, it's easier to end up with a melodically pleasing result, and it could be tailored to harmonize for different genres.

Built With

• krita
• krita-libkis
• pyqt
• python
• qt