MRTodo: Synchronizing Paper and Digital To-Do Lists in MR

Inspiration

We find people still prefer writing their to-do lists on paper. A notebook and pen are frictionless - no battery, no loading time, no notifications. People can write anywhere, anytime. Research finds that Handwriting also helps people think more clearly and feel more in control of their plans.

But over time, a recurring problem arises.

As tasks accumulate, the notebook becomes cluttered. It’s hard to tell which tasks are completed, which are pending, and how to reorganize priorities. Even worse, the handwritten tasks are disconnected from the digital task management systems.

People have to manage the same tasks twice.

This fragmentation between paper and digital workflows inspired MRTodo.

What it does

MRTodo enables bidirectional synchronization between handwritten paper and digital task management using Mixed Reality.

Users write naturally on paper with a modified Logitech MX Ink stylus. The system captures pen trajectories, recognizes handwritten tasks using a vision-language model, and converts them into structured digital task objects.

Once digitized, tasks can be:

Sorted and filtered digitally
Updated and reorganized
Synchronized back onto the physical notebook

To maintain alignment between dynamic digital lists and fixed handwritten entries, MRTodo introduces two interaction techniques:

Visual Linking – connects a selected digital task to its corresponding handwritten entry.
Adaptive Overlay Shifting – repositions digital overlays to maintain spatial consistency.

This keeps paper and digital representations continuously aligned.

How we built it

MRTodo is built in Unity and runs on Meta Quest 3.

Key components include:

Hybrid Stylus Design
We retrofitted the Logitech MX Ink stylus with a metal-alloy tip, allowing users to write on real paper while preserving 6-DoF tracking and pressure sensing.
Stroke Capture & Recognition
Pen trajectories are captured in real time. A vision-language model converts handwritten content into structured JSON representations (including task content, deadline, priority, etc.).
Paper Tracking & Spatial Anchoring
The notebook is tracked using ArUco markers to ensure digital overlays remain spatially aligned with physical content.
Cloud Synchronization
Structured task data is stored and updated digitally, allowing real-time reorganization.

The system maintains a continuous feedback loop between paper and digital space without physically altering the notebook.

Challenges we ran into

One of the biggest challenges was digital-to-paper synchronization.

Digital task lists are dynamic — they can be sorted or filtered instantly. But handwriting on paper is spatially fixed. When digital order changes, the physical layout does not. Maintaining correspondence between the two required careful interaction design.

Handwriting recognition was another challenge. Vision-language models sometimes reinterpret text rather than transcribe it literally. Balancing accuracy, latency, and user control required iterative prompt engineering.

Hardware constraints were also a factor. Extended MR headset use introduced physical fatigue, highlighting the importance of lightweight interaction design.

Accomplishments that we're proud of

Successfully enabling real-time bidirectional synchronization between paper and digital tasks
Designing two novel alignment strategies for hybrid interaction
Preserving the tactile authenticity of handwriting while adding digital intelligence
Demonstrating that MX Ink can serve as a bridge between physical and spatial computing workflows

Most importantly, we created a system that respects existing paper-based habits rather than replacing them.

What we learned

We learned that paper is not obsolete - it offers cognitive and tactile benefits that digital systems still struggle to replicate.

We also learned that hybrid systems are fundamentally about alignment - spatial alignment, cognitive alignment, and workflow alignment.

Generative AI is powerful, but it must be transparent and controllable when dealing with personal information like handwritten notes.

Most of all, we learned that innovation does not always mean replacing the old - sometimes it means intelligently augmenting it.

What's next for MRTodo

Next, we want to:

Improve handwriting transcription fidelity using more specialized models
Explore lighter-weight MR hardware to improve comfort
Support multi-page notebooks for long-term usage
Integrate with existing productivity ecosystems (calendar, email, task apps)
Investigate applications beyond task management, such as education and healthcare logs

Our long-term vision is to make persistent physical artifacts like paper first-class citizens in spatial computing environments.