Combining our passions for music and drawing, we were inspired to develop a creative medium that bridged the two together. We finally arrived at the idea of an app that converts drawings into music. Although the concept sounded intimidating at first, we were super excited to begin!
What it does
Simply put, TinkerTunes transforms drawings into music. With multiple colors to choose from, custom brush sizes, and features like undo/clear, users can unleash their creativity to draw anything they’d like. TinkerTunes has three musical environments: piano, guitar, and drums. When switched to a specific environment, each color corresponds to a different musical note (ex. Piano: Red for C, Orange for D). Once the user finishes their drawing, they can transform their artwork into a unique 15-second tune. This is done with a function that analyzes the artwork’s color pixels, playing the notes that correspond with each color.
How we built it
We built TinkerTunes using Swift and Xcode. More specifically, we used Swift to scan the canvas’ color pixels in a left to right, top to bottom order, to play notes that correspond with the colors on the canvas. We installed a canvas with drawing capabilities through CGContext and Canvas view. From there, we used RGB components to identify colors on the canvas. To play tunes from the drawing, we used AVFoundation’s AVAudioPlayer to output instrument sounds to the device.
Challenges we ran into
With thousands of pixels on a screen, it was challenging for us to find a performance-efficient method of representing every pixel of color with its musical sound. However, after many hours of experimenting, we were able to optimize the app’s performance by creating a system of pixels with larger sizes and limited repetition, compressing each tune to around 15 seconds long. Furthermore, we ran into several difficulties playing the musical sounds, as our loop that iterates over every pixel was running far too rapidly, leaving only a fraction of a second to play a musical note. To combat this, our team used a sleep function to delay the quick iteration through each colored pixel, leaving adequate time for every musical note to be heard.
Accomplishments that we're proud of
Launching a far-fetched idea into a functional app, we’re proud of taking a vague aspiration from the start and carrying it across the finish line. Despite facing numerous challenges along the way, we remained resilient and continued to persevere.
What we learned
In our making of TinkerTunes, we learned how to create a canvas using Swift and how to play audio using AVFoundation. Furthermore, we learned how to more efficiently use online resources like Stack Overflow and YouTube tutorials to learn new skills.
What's next for TinkerTunes
In the future, we hope to expand our audio library to include more instruments like flutes, trumpets, and xylophones. We hope to refine our scanning algorithm to decrease repetition in tunes by eliminating groups of pixels from the sound output loop who fall under a similar pattern. Additionally, we plan to add the ability for users to save their drawings and tunes into a png and mp3 file respectively, to keep and share their artwork beyond this app!