• 

• 

• 

• 

• 

• 

mirrons

We have been painting motion for thousands of years. Cave paintings imply movement with repeated limbs, photographers use long exposures, painters suggest gesture with brushstrokes. All of those are interpretations by another human looking at movement.

This is project makes paintings by motion itself. Every stroke in this piece comes from a computer estimating how every point in the camera image moves through time. The artwork is literally generated from vectors that exist only because a computer is analysing video frame by frame.

That makes it an artwork that simply could not exist without technology. Here's the story...

I was annoyed

Webcam particle demos are not novel. I'd seen many, but they all felt the same, and it took me a while to work out why: it's because they use frame differencing. They subtract this frame from the last one, threshold it, and get a mask of "stuff changed here". Then you scatter particles inside the mask.

The problem is that a frame difference has no direction in it. It knows your hand was there, but not that your hand went, say, to the left. So, every one of those demos ends up with particles that fizz... where you are instead of moving the way you moved, and you can feel the difference immediately even if you cannot name it.

I wanted my art to highlight the gesture itself, the direction and the speed instead of a silhouette of where it happened.

Optical flow to the rescue

Optical flow is the math that fixes it! The standard approach is Lucas-Kanade, from 1981, and the idea is genuinely lovely once it clicks in your mind. Let's assume a patch of image keeps its brightness as it moves. Then for a small displacement \(u,v\) over one frame:

$$ I_x u + I_y v + I_t = 0 $$

... where \(I_x\) and \(I_y\) are "how fast brightness changes" across the image and \(I_t\) is "how fast it changed"... "in time". One equation, two unknowns, so a single pixel cannot tell you anything. But if you assume a whole window moves together you get one equation per pixel in that window, and you can solve the overdetermined system by least squares:

$$ \begin{bmatrix} \sum I_x^2 & \sum I_x I_y \ \sum I_x I_y & \sum I_y^2 \end{bmatrix}\begin{bmatrix} u \ v \end{bmatrix} = -\begin{bmatrix} \sum I_x I_t \ \sum I_y I_t \end{bmatrix} $$

That 2x2 on the left is the structure tensor. So I invert it, I get my velocity, and I'm done, right? No, because as I found out, it does not always invert. There is a matrix of zeros if you point the camera at a blank wall because there is no gradient anywhere. If you point it at a straight edge you can see motion across the edge but not along it, because sliding along an edge looks identical to not moving. That is the aperture problem, and it is the same reason a barber pole looks like it moves upward forever.

The fix I liked is to look at the smaller eigenvalue \(\lambda_{min}\) of the structure tensor. If it is near zero the window is ill conditioned and the answer is garbage, so throw it away. I use \(\lambda_{min}\) directly as a confidence weight, and cells below a threshold get discarded rather than believed. This is basically the Shi and Tomasi corner criterion from 1994, and I found it by reading about corner detection while trying to work out why my flow field kept exploding against a plain wall.

Where was I wrong?

I got that working and it looked great, and then I waved my arm properly and the whole thing fell apart. The brightness constancy equation above is a first order Taylor expansion. It only holds for small displacements. My window was 13 pixels across, so a hand moving further than about 6 pixels per frame broke the assumption completely, and it broke silently, which is worse. It did not error, it just returned confident nonsense.

I only found the size of the problem because I wrote a test that translated a synthetic image by a known number of pixels and asked the solver to recover it, without telling it the answer. That test is test/flow.js and it was the most useful hour I spent on the whole thing:

gesture speed	error, single level
240 px/s	0.3 %
480 px/s	13.3 %
1440 px/s	78.7 %

Fine when I moved slowly for the camera. Useless for actually making art, because nobody dances slowly enough for a naive solver. The fix is Bouguet's pyramid scheme. Build a stack of progressively halved images. Solve on the tiniest one first, where a fast gesture only moves a pixel or two and the linearisation holds fine. Take that answer, double it, use it to warp the previous frame at the next level down, and solve again for whatever is left over. Repeat to full resolution. Same maths, run four times, with the hard part already removed before the accurate level ever sees it.

gesture speed	single level	4 level pyramid
240 px/s	0.3 %	0.0 %
480 px/s	13.3 %	0.0 %
960 px/s	-	0.5 %
1440 px/s	78.7 %	0.0 %
1920 px/s	-	2.0 %

Whole vision stage costs 3.4 ms a frame. Now one case in that test kept failing and I nearly rewrote the solver over it. On a sine grid, a big displacement came back pointing 160 degrees backwards. I stared at that for a long time before working out it was correct behaviour. A repeating pattern shifted by almost one full period genuinely does look like a small shift the other way. There is no information in the image to tell those apart. The ambiguity belongs to the picture, not the algorithm. So the test now checks extreme speeds against band limited value noise, which is what a real room looks like, and the same displacements come back within 0.2 degrees. I left the periodic case in as a comment because it taught me something.

From flow to sculpture

Once I had a velocity field, the painting part was almost easy. Particles get emitted where the field is strongest, in proportion to how much is moving, so standing still costs nothing. They get advected by the field, and they deposit onto a canvas that never clears. That last bit is the whole idea. It is a long exposure of your gestures, insteaad of just a live view. Nothing on screen is where you are, it is where you have been.

Then I wanted it to stop being flat. Two surfaces accumulate at the same time: one colour buffer, and one half resolution buffer holding the same strokes in white. That second one is a height field. Read it back, take central differences to get the slope in each direction, and you have a surface normal:

$$ \mathbf{n} = \frac{(-\partial h/\partial x,\; -\partial h/\partial y,\; 1)}{\lVert \cdot \rVert} $$

Light that with Blinn-Phong and your gestures become a carved thing with a light falling across it. Because both surfaces fill up simultaneously, the carve checkbox can be flipped at any moment and the relief is already sitting there waiting.

Palettes carry material properties rather than just colours, which I did not expect to need. A ramp that looks good as glowing pigment looks like plastic when you light it. So each palette also has a specular colour, a shininess exponent and an ambient level, and patina reads as oxidised bronze while plaster reads as dry cast gypsum on a light ground.

The variable-tempo piano

Lastly, I wanted music that was generated rather than looped, without hardcoding a scale.

It turned out to be simpler than expected. Hold one seventh chord at a time as a set of intervals. To move on, pick a new root by an interval drawn from a short weighted list that is mostly \(+5\) semitones, which is down a perfect fifth. That single choice is most of what makes something sound like jazz, because the circle of fifths is what ii-V-I motion actually is. Sprinkle in \(+6\) for a tritone substitution and \(+2\) for a step slide.

The bit I like is that when the chord changes, the next note played is whichever tone of the new chord sits closest in pitch to the note just played. That is voice leading, expressed as a nearest neighbour search over about twelve numbers. The melody keeps walking by a semitone or two while the harmony moves underneath it, and it sounds intentional even though nothing is written down anywhere.

I wrote a test for this too: test/music.js runs the piano on a fake clock and checks every note belongs to its current chord, that roots only move by declared intervals, and that the melody does not leap at chord changes. The first run failed on the last one. It turned out I picked the nearest linking tone and then immediately stepped past it before playing anything, so the linking was computed and then thrown away. Median jump across a chord change was 4 semitones, worst was 13. Playing the linked tone directly brought the median to 1 semitone and the worst to about 5. You can hear the difference, but I would never have found it by ear.

Tempo follows how much the field is moving, 58 bpm when you are still up to 136 when you are throwing your arms around.

Ghosts

This feature I added lets you record a gesture and it replays as its own painter, so you end up working next to your earlier self. The samples are normalised to 0..1 so a ghost recorded on a laptop replays correctly on a projector, and they export to a small JSON file with motion vectors only, no imagery. Someone else can load your ghost file and paint over it. That is the honest version of collaborative for an art piece with no server: not live co-presence, but other people's gestures really do pile onto the same surface.

What I learnt

That the interesting problem was not the particles. Anyone can make particles. The interesting problem was that a gesture has a direction and almost every tool for reading a camera throws that away in the first step.

That "it looks right" is not the same as "it is right". Both times I wrote a test that fed the system a known answer and asked it to recover the answer, it found something that was quietly wrong and that I had been looking at for hours without noticing. The flow was 79 % off at speed. The music computed a voice leading and then discarded it.

That a failing test is not always a bug. The backwards sine grid was the algorithm behaving correctly on an ambiguous input, and the right response was to fix the test rather than the code. Working out which of those two you are looking at took longer than either fix.

Built with

No dependencies, no build step, no framework, no API keys, no server, no third party assets of any kind. One HTML file, 66 KB, runs offline from disk. Canvas 2D, Web Audio, getUserMedia and MediaRecorder are all browser native. Particles are a structure of arrays over Float32Array, allocated once, recycled in place, so there is no garbage per frame.

The camera stream is drawn into an offscreen canvas, reduced to 160x120 greyscale and thrown away every frame. Only the previous frame is kept, because the temporal gradient needs it. The page makes no network requests at all after it loads, so there is nothing to send anywhere and nowhere to send it.

Lastly, I host the online version as a GitHub Pages project site on my own domain, but the application itself is completely static and works without any backend.

Why this could not exist without technology

None of the finished images exist in front of the camera.

They emerge from quantities humans never perceive directly: image gradients, optical flow, confidence estimates and accumulated motion over time.

The sculpture is made from movement rather than pigment. The music is generated from the same motion field that shapes the image. Ghosts preserve trajectories instead of recordings.

Without computation there is no artwork to look at—only an ordinary webcam pointed at a person waving their arms.

Built With

• blinn-phong
• canvas
• computer-vision
• creative-coding
• eigenvalues
• float32array
• generative-art
• getusermedia
• html5
• image-pyramid
• javascript
• lucas-kanade
• mediarecorder
• music-theory
• normal-mapping
• optical-flow
• particle-system
• procedural-music
• simplex-noise
• structure-tensor
• vanilla-js
• webaudio
• webrtc
• zero-dependencies