AirTouch

AirTouch rethinks the touchscreen as a spatial interface instead of a piece of hardware.

Inspiration

I recently bought my first MacBook. I loved the build quality, the trackpad, the display, and how polished everything felt, but I was genuinely disappointed that I could not touch the screen.

Since I was already experimenting with hand tracking on Snap Spectacles, I started wondering: what if the laptop did not need a physical touch sensor? Spectacles already know where my fingers are in 3D space, so maybe they could figure out when I was touching the display.

That slightly stubborn thought became AirTouch.

What it does

AirTouch lets me use a regular MacBook or Windows laptop like a spatial touchscreen while wearing Snap Spectacles.

I start by pinching the four corners of the display:

Top Left -> Top Right -> Bottom Left -> Bottom Right

This creates a virtual surface aligned with the real screen. After calibration:

Moving my finger over the display moves the cursor
Touching the virtual screen clicks
Touching and moving performs a drag
Lifting my finger releases the mouse
Moving my index and middle fingers scrolls
Pinching can also be used as a click-and-drag fallback

There is no touchscreen overlay or laptop modification. The screen itself has no idea that I touched it. Spectacles understand the interaction and the desktop companion translates it into mouse input.

How I built it

The Spectacles Lens uses Spectacles Interaction Kit to track my index and middle fingertips in world space.

After calibration, AirTouch projects each fingertip onto the virtual screen and converts its position into normalized coordinates:

u = horizontal position from 0 to 1
v = vertical position from 0 to 1

Those coordinates travel over a local WebSocket connection to a Python desktop companion.

Spectacles Hand Tracking
        ↓
Four-Corner Calibration
        ↓
Spatial Screen Projection
        ↓
WebSocket Pointer Packets
        ↓
Desktop Companion
        ↓
macOS Quartz / Windows User32
        ↓
Cursor, Click, Drag, and Scroll

The companion converts the normalized coordinates into screen pixels and creates native mouse events using Quartz on macOS or User32 on Windows.

I developed AirTouch end to end with Codex as my implementation partner. I used it while building the Lens logic, calibration, gesture handling, networking, desktop companion, Windows support, smoothing, testing tools, and documentation.

For the latest accuracy improvements, I used Codex powered by GPT-5.6 to inspect the existing projection code and improve it. One important discovery was that I was collecting four calibration corners, but the original projection math only meaningfully depended on three of them. I replaced that with a bilinear quadrilateral projection that uses every corner, validates bad calibration shapes, and maps the complete screen more accurately.

Challenges I ran into

The hardest part was finding a good balance between jitter, accuracy, and latency.

Increasing smoothing made the cursor stable but also made it feel like it was chasing my finger. Reducing smoothing made it responsive, but selecting small buttons became difficult.

I ended up combining adaptive smoothing, small movement deadzones, limited prediction, desktop-side click anchoring, and separate touch and release thresholds.

Calibration also turned out to be more complicated than just saving four points. Hand-tracked corners are never perfectly rectangular, so small errors became much more noticeable near the bottom-right of the display. Using all four corners in the projection made a real difference.

Networking caused one particularly silly debugging session. I initially forgot to include the port in the WebSocket URL. I also had to remind myself that 127.0.0.1 on Spectacles refers to the glasses, not my laptop.

Accomplishments I'm proud of

Making an ordinary laptop respond to spatial touch
Supporting hover, click, drag, release, and two-finger scrolling
Adding direct screen-plane touch without requiring a pinch
Building accurate four-corner calibration
Supporting both macOS and Windows
Generating a visible calibration plane with MeshBuilder
Adding safety releases when my finger leaves the screen
Creating a deterministic demo that works without Spectacles hardware

The first time the MacBook cursor followed my fingertip across the screen felt a little ridiculous, but also genuinely magical. It was the moment the project stopped feeling like an experiment and started feeling like an interface.

What I learned

The biggest thing I learned is that touch does not have to be a physical feature of a display.

It can simply be a relationship between:

a finger + a screen + a spatial coordinate system

I also learned that good spatial interaction requires much more than gesture detection. Calibration, latency, smoothing, safety, and visual feedback all matter.

Most importantly, this project changed how I think about wearable computing. Glasses do not always need to replace the devices around us. They can give new abilities to the devices we already own.