PhantomTouch: Turning Any Surface Into a Touchscreen

Inspiration

What if the way we interact with computers could adapt to us, instead of forcing us to adapt to them?

We use keyboards, mice, and trackpads every day without thinking twice. But for people with limited hand mobility, motor difficulties, or other physical disabilities, these traditional input devices can make everyday tasks much harder.

We wanted to explore a different approach: what if the desk in front of you could become your computer's input device?

That's what inspired PhantomTouch. We wanted to build something technically challenging, but also something that could make computers more accessible without requiring expensive specialized hardware.

What It Does

PhantomTouch turns an ordinary physical surface into an interactive digital interface using just your laptop's camera.

Using computer vision and hand tracking, PhantomTouch detects finger movements and translates them into digital interactions.

Our system is designed around three ways of interacting:

• Trackpad Mode: Move a cursor, click, drag, scroll, and navigate applications using hand gestures.
• Drawing Mode: Draw on a physical surface and translate those movements into digital strokes.
• Writing Mode: Capture handwritten gestures, recognize text, and insert it into computer applications. The longer-term goal is to let users interact with their entire computer rather than remain confined to one application.

Accessibility is central to the design. We are building adjustable gesture sensitivity, larger controls, alternative activation methods, and one-handed interaction options so users can personalize the experience to their abilities.

No gloves, wearables, or specialized sensors are required.

How We Built It

We used MediaPipe for real-time hand landmark detection and OpenCV for processing camera input and mapping finger positions to a calibrated physical surface.

Through perspective transformations, we convert positions detected by the camera into corresponding coordinates in the digital workspace.

We also developed gesture recognition to distinguish between finger movement and intentional actions, using techniques such as smoothing, debouncing, and gesture thresholds to reduce accidental interactions.

For the interface, we used web technologies to create a workspace with calibration controls, tracking feedback, accessibility settings, and interactive tools.

We also implemented a real-time collaboration system using Convex, allowing users to share drawings and manipulate objects across connected sessions, and built a multiplayer experience using SpacetimeDB.

Our system-wide macOS interaction work uses a native input bridge designed to translate gestures into mouse and keyboard events with explicit user permission.

Challenges We Faced

The biggest challenge was making finger movements feel as intentional and precise as a real input device.

A webcam isn't a touchscreen. It doesn't directly know when a finger touches a surface, and even small movements or tracking errors can cause unintended interactions.

Early in development, our drawing system was extremely sensitive. Movements that felt small in the real world created exaggerated, shaky lines on the screen.

This pushed us to rethink the interaction model. Instead of focusing only on drawing, we began building a more flexible system with configurable sensitivity, deliberate gestures, and native computer controls.

We also had to account for camera positioning, perspective distortion, varying lighting conditions, and the performance requirements of real-time tracking.

What We Learned

We learned that building an alternative input device involves much more than detecting a hand.

The difficult part is turning imperfect computer vision data into interactions that feel stable, predictable, and comfortable.

We also learned how computer vision, geometric transformations, real-time systems, and operating system APIs can work together to create entirely new forms of human-computer interaction.

Most importantly, this project changed how we think about accessibility. Instead of expecting everyone to use a computer in the same way, why not build interfaces that adapt to different users?

What's Next?

We want to continue improving system-wide computer control, precision handwriting, gesture customization, and accessibility.

Future improvements include better gesture recognition, personalized movement calibration, more accurate handwriting recognition, and testing with people who use alternative input devices.

Our long-term vision is to make computers easier to interact with, regardless of someone's physical capabilities or the equipment they own.

Technology should adapt to people, not the other way around.

Built With

• ai
• computervision
• convex
• fastapi
• fetch.ai
• gesturerecogn
• handtracking
• javascript
• mediapipe
• next.js
• numpy
• opencv
• python
• react
• spacetimedb
• tailwindcss
• typescript
• websockets