Penultimate

🚀 Inspiration

Most laptops have incredible displays, but they lack native touchscreen/stylus support. Instead of accepting that limitation or buying an expensive secondary tablet, we wanted to see if we could bridge that hardware gap entirely through clever software and cameras.

What it does

Penultimate utilizes a Raspberry Pi equipped with two camera modules to monitor a laptop screen. It watches the pen, calculates its coordinates in real-time through computer vision, and then transmits that data over to your laptop for rendering, In doing so, it ensures minimal lag throughout the processing.

In terms of what it can do on your laptop, Penultimate is equipped with three modes. Cursor control allows the pen to move the cursor and cause a click when pressed, which allows Penultimate to be software-agnostic. Annotation allows the pen to easily draw or diagram on top of any app through an overlay. Notes provides a complete note-taking experience similar to a tablet.

How we built it

Hardware

We paired a Raspberry Pi 5 with two detachable Camera Modules to create the tracking system. They are easily attached to the corners of the laptops.
We built and 3D-printed a custom pen featuring an LED located at the writing tip. It is completely wireless and powered by batteries.
The Raspberry Pi acts like an external processing hub, and communicates with the laptop over a local UDP network connection.

Computer Vision

We used Picamera2 to capture synchronized frame pairs, which bypasses slower libraries to minimize processing latency. We utilized color filtering and blob detection to pin down exactly where the tip is in each frame.
A calibrated perspective transform then warps those camera coordinates onto the screen, correcting for the angle the cameras are looking from.
To tell when the pen is actually touching, we watch how the LED blooms and blurs as it nears the surface, and register a tap once that holds for a few frames once that is detected.

Software

The laptop runs a desktop app made with Tauri that listens for the pen data over the network and turns it straight into cursor movement, so the pen just drives the Mac like a touchscreen would. A Rust core handles the high-frequency work, which also parses the incoming pen packets.
The notebook is programmed using React, creating a user-friendly interface to allow easy note-taking.

Challenges we ran into

Reducing latency as much as possible, which appeared throughout the system
Making the computer vision as accurate as possible
Mapping the warped 2D camera perspectives into flat coordinates, in which we tried various approaches like using a homography matrix and planar PnP pose estimation.

Accomplishments that we're proud of

The calibration routine is very fast, only needing around one second without the user needing to input anything!
The stylus is complete and surprisingly ergonomic (despite fitting quite a few hardware components inside).