DALL-E Picture Frame

AI Generated Trees
Space surfers
The team
Wiring

Inspiration

As first-time hackers, our goal for TreeHacks was to do something fun but manageable. We were also itching to get our hands on some hardware and tinker with some microcontrollers. During the opening ceremony we were hearing about all the latest advancements from OpenAI and other Artificial Intelligence companies, and we loved image generators, so we thought we could leverage some _ fancy stable diffusion _ to make our project do something super cool. And so, the idea of the LED matrix picture frame, or "Elly Dee" as we like to call her, was born.

What it does

Our LED matrix first reads in a prompt from the user for an AI-generated image, just like you would input in any AI image generator, and uses OpenAI's DALL-E API to generate some fantastic pictures. It then uses some Python magic to trasnform the images to a lower resolution and resize them so that it can be written to our Adafruit LED Matrix. The picture data is encoded and sent wirelessly to our Arduino, where it writes the AI's creation to the LED matrix!

How we built it

The DALL-E interfacing and image manipulation was done in Python using OpenCV. For the hardware setup, we used an Arduino Mega, a 32x64 Adafruit LED Matrix, and an Adafruit ESP-32 for Wi-Fi capabilities. Although it was recommended to use a "hat" that does all the hardware interfacing to the LED Matrix, we actually wired everything ourselves so we could pin the ESP-32 and the Matrix how we wanted.

Challenges we ran into

Our biggest challenge was the curse of the TreeHacks2Ghz WiFi network. We were using the UDP protocol to send bytestreams of image data to our Arduino, and packets were being lost left and right, data corrupted. We still didn't fully figure out why this is happening, maybe because of a very overworked network, or maybe because we just dont know how this stuff works and are doing something terribly wrong. We worked out some quick fixes but in the future we'd probably want a better way to transmit the data.

Also, hardware! We got what we wished for, which was complications left and right with wiring, power sources, and stuff just breaking. There are definitely pros and cons to working with hardware.