Computertop Desk

Inspiration

Ideas for interactions from:

http://paperprograms.org/
http://dynamicland.org/ but I wanted to go from the existing computer down, rather from the bottom up, and make something that was a twist on the existing desktop: Web browser, Terminal, chat apps, keyboard, windows.

What it does

Maps your Mac desktop windows onto pieces of paper + tracks a keyboard and lets you focus on whichever one is closest to the keyboard. Goal is to make something you might use day-to-day as a full computer.

How I built it

A webcam and pico projector mounted above desk + OpenCV doing basic computer vision to find all the pieces of paper and the keyboard.

Challenges I ran into

Reliable tracking under different light conditions.
Feedback effects from projected light.
Tracking the keyboard reliably.
Hooking into macOS to control window focus

Accomplishments that I'm proud of

Learning some CV stuff, simplifying the pipelines I saw online by a lot and getting better performance (binary thresholds are great), getting a surprisingly usable system.

Cool emergent things like combining pieces of paper + the side ideas I mention below.

What I learned

Some interesting side ideas here:

Playing with the calibrated camera is fun on its own; you can render it in place and get a cool ghost effect
Would be fun to use a deep learning thing to identify and compute with arbitrary objects

What's next for Computertop Desk

Pointing tool (laser pointer?)
More robust CV pipeline? Machine learning?
Optimizations: run stuff on GPU, cut latency down, improve throughput
More 'multiplayer' stuff: arbitrary rotations of pages, multiple keyboards at once