Sound Lab in Action
There's this really awesome musician Jerobeam Fenderson who makes music that has a music video when viewed on an x-y oscilloscope.
We thought this was really cool, and wanted to learn how to make our own oscope demos!
What it does
We created a few different applications that allow you to use the oscope in unique ways.
Sound lab is a gui that lets you explore how sending different sound signals of your own design to the left and right audio channel produces different images in the oscope.
Drawing extends the idea of sound lab, but lets you draw any image using your mouse and keyboard, and the image will show up on the scope.
Pong lets you play pong on the oscope!
Arbitrary Images (Attempted)
We tried to use edge detection to get arbitrary images to display on the oscilloscope.
How we built it
In order to draw on an oscilliscope, you need to put the scope into "XY" mode. This mode takes the audio output from two channels (left and right), and plots the channels against eachother. For example, if you had a cosine wave on the x channel and a sine wave on the right channel, the oscope would display a circle.
You can kind of think of the sounds being created by the laptop as parametric functions with a timestep, so with all of these projects our goal was to produce audio on the fly based on these parametizations.
To make developing so many apps easy, we created a library that can take any line and produce a waveform that looks like that line on the oscope. By playing the noise for a bunch of lines in succession, you can draw any arbitrary shape.
To produce these waveforms, you just take the normal line equation y=m(x-x1)+y1, and parametize it. So on the y channel your waveform would be the function y(t) = (m(x-x1)+y1) * sin(t) and on the x channel you would output x(t)=(m(x-x1)+y1) * cos(t).
This library powers the Pong game, we just specified the two paddles and ball using these lines, and created waveforms that we can modulate to move them around on the screen.
We also used this library for the drawing app, but with modifications. It would be way too slow to generate a sound for each pair of points in the drawings the use makes, (if we take too long to draw the screen, you can an awful flickering effect). To combat this, we used a line combining algorithm to make an approximation of the drawing entered by the user, reducing the number of points needed significantly.
Challenges we ran into
The oscope refresh rate is lower than we expected, so the sampling causes sharp edges when we produce square waves. This can be seen in pong when looking at the corners of the paddles, they are slightly rounded because of this effect. We also needed to run a simplification algorithm on the lines we were trying to render in the drawing app.
Also, my Macs sound driver does automatic output equalization to stop your ears from being destroyed by sounds that wouldn't normally be conducive to listening to, but as a result, as the position of content on the screen changes, the oscope position drifts when my mac is used. This took us a long time to figure out, as we thought it was our softwares fault. Switching to the surface book fixed our drifting issue.
Finally, the image app proved to be difficult. Edge detection is fairly straightforward, but producing a set of lines to display and scheduling them in an optimal order proved to be very difficult. We were able to get some stuff to show up on the display but it looked like scribbles.
Accomplishments that we're proud of
It's extremely satisfying to play a sound on your laptop, and have it by rendered as the videogame pong on an oscope. We're really proud of all of the applications we were able to complete, and we think we made really good progress.
What we learned
We learned a lot about how sound is generated, and how aliasing / sampling frequency effects the oscope output.
What's next for Oscope Drawing
We'd love to get a higher frequency scope and finish the webcam app.