Ideas for interactions from:
- http://dynamicland.org/ but I wanted to go from the existing computer down, rather from the bottom up, and make something that was a twist on the existing desktop: Web browser, Terminal, chat apps, keyboard, windows.
What it does
Maps your Mac desktop windows onto pieces of paper + tracks a keyboard and lets you focus on whichever one is closest to the keyboard. Goal is to make something you might use day-to-day as a full computer.
How I built it
A webcam and pico projector mounted above desk + OpenCV doing basic computer vision to find all the pieces of paper and the keyboard.
Challenges I ran into
- Reliable tracking under different light conditions.
- Feedback effects from projected light.
- Tracking the keyboard reliably.
- Hooking into macOS to control window focus
Accomplishments that I'm proud of
Learning some CV stuff, simplifying the pipelines I saw online by a lot and getting better performance (binary thresholds are great), getting a surprisingly usable system.
Cool emergent things like combining pieces of paper + the side ideas I mention below.
What I learned
Some interesting side ideas here:
- Playing with the calibrated camera is fun on its own; you can render it in place and get a cool ghost effect
- Would be fun to use a deep learning thing to identify and compute with arbitrary objects
What's next for Computertop Desk
- Pointing tool (laser pointer?)
- More robust CV pipeline? Machine learning?
- Optimizations: run stuff on GPU, cut latency down, improve throughput
- More 'multiplayer' stuff: arbitrary rotations of pages, multiple keyboards at once