Ellsworth Kelly's Hidden Agenda

Inspiration

I began this project by exploring the halls of this excellent museum, hoping to take inspiration from the artworks themselves.

I reckoned that most teams would either make geo-spatial ("Pokemon Go") games which explored the building, but did not encourage deeper understanding of the art on display, or social applications which allowed users to share their opinions with each other, but not to have an intimate and meaningful experience with a given artwork that could only happen with technology.

So, I decided wanted to make something that:

• Embraced true hacker purpose and aesthetic
• Gave the player a meaningful connection to a particular artwork
• Used the physical space of the museum
• Stands alone as art-object
• Poked a little bit of fun at the event itself

While exploring, I noticed that room 175 was completely devoid of color, and that Ellsworth Kelly's works looked almost exactly like the Augmented Reality Matrix Codes used by the AR.js toolkit.

And so, the seed was planted.

(I also assumed that it was impossible to actually win the contest as an individual, which was actually quite liberating. Hacking should never be about money. Hacking for personal amusement is the only true hacking.)

What it does

There are two components: an augmented reality sculpture and viewer, and a mobile exploit kit and malware sample.

The scanner is able to use a hidden barcode pattern that Kelly accidentally put in his painting to display a digital sculpture in augmented reality. Discovering this sculpture triggers a callback to the mobile exploit kit, which then infects and executes.

Watch the attached video to know why.

PLEASE NOTE that visiting the website is completely safe for the general audience! For obvious reasons, only the augmented reality component of this project will be shared with the public. I will only share the hacking tools with those who are capable of handling them responsibly.

How I built it

JS/AR.js/ARToolkit, Python/Flask/Zappa, and some stealing from Metasploit. The music is made in Ableton Live. The voxel art is sculpted in Magica Pixel. The video was made using ffmpeg, GIMP and iMovie.

Challenges I ran into

Time. I didn't realize this hackathon would have a video component, which adds a quite a heavy production requirement to an already ambitious project.

I also worked entirely alone, which I don't normally like to do at hackathons, but this project ended up being too political and too tied to my own personal history as an artist to allow for meaningful collaboration with strangers.

Still, that meant that I had to do all the programming, 3D modelling, music, design, video editing and writing! Phew.

Accomplishments that I'm proud of

This is the world's first computer virus which uses a fine art object as an infection vector. I think that's kinda cool.

I was hoping to work with another participant on the 3D model design, but that never happened so I had to do that myself too. That's probably the part I'm the least proud of. It's not terrible though.

What I learned

I learned quite a lot about a man named Ellsworth Kelly!

What's next for Ellsworth Kelly's Hidden Agenda

Sell it to the highest bidder, naturally.

Built With

• assembler
• c
• javascript
• python
• ruby