Inspiration
Last year, as a senior at Duke Engineering, I saw a senior design project that stayed with me. The team used computer vision and a custom glove to help people who needed to communicate through sign language but did not have a full range of motion in their hands.
I admired how they designed around the abilities of the people they wanted to help. Their work also raised a larger question for me: how many people have circumstances that do not fit the assumptions of an existing interface?
People do not fit neatly into categories. Two people with the same diagnosis may have very different movements available to them. Someone might comfortably raise an eyebrow, another might control a hand, and another might rely on small eyelid movements. Those abilities can also change with fatigue or over time.
Existing communication tools offer valuable forms of customization. I wanted to explore how far that personalization could extend—into the interaction itself.
What if someone could create a communication interface around the movements they can make, instead of having to find an interface they can adapt to?
That question became Relay.
My ambition is a universally adaptable communication platform: a place where users, loved ones, and creators can invent ways to communicate, use them immediately, and share their ideas with others.
What it does
Relay is a browser-based platform for creating, using, and sharing movement-controlled communication environments through an ordinary webcam.
In my recorded demo, I wrote whole sentences using only movements of my eyelids.
Those movements acted as controls for composing text. I could choose what to say, build a message, and have it spoken aloud. That demonstration made the project feel much more meaningful to me: a small movement could become access to an entire alphabet and an open-ended conversation.
Relay connects three spaces:
User space
Users can record their resting behavior and teach Relay comfortable facial expressions, hand poses, or short movements. Those signals can navigate an interface, select a character, insert text, delete, undo, pause, or request speech.
A user does not need a different gesture for every letter. Two signals can navigate and select across a full keyboard; one reliable selection signal can work with automatic scanning.
Different environments share the same message and editing workflow, so users can change how they communicate without starting over.
Creator space
Creator lets someone build the interface those movements control.
A creator can place characters and phrases on a canvas, organize groups, connect measured movements to interface properties, and define how selections happen. Drawing surfaces can use supported local recognition models or personally taught symbols.
For example, a hand-controlled dial can be assembled from individual letters, a circular arrangement, a hand-rotation mapping, a fixed target, and a hold-to-select rule. Change the arrangement and mapping, and the same components can become a sliding character strip.
The person who uses an environment does not have to be the person who creates it. A loved one, caregiver, accessibility practitioner, or another user can contribute an idea and tailor it to someone’s circumstances.
Environment feed
The public feed lets people browse, try, remix, publish, and vote on environments.
This turns a personal experiment into something another person can adapt. Sharing the environment does not share the original user’s face or hand calibration.
Relay includes six starting environments: a personalized gesture keyboard, a two-hand finger grid, air handwriting, a circular letter path, a swipe-word keyboard, and phrase shortcuts. After developing those interfaces, I generalized their interaction ideas into reusable Creator components and editable templates.
The examples demonstrate possibilities. The creator makes room for ideas I have not thought of.
How I built it
I built Relay as a solo project using JavaScript, HTML, and CSS, with computer vision and recognition running locally in the browser.
I separated the system into three responsibilities:
- Perception: What movements can the camera observe?
- Interpretation: What signal does that movement represent?
- Interaction: What should that signal do in the current environment?
MediaPipe provides pretrained face and hand tracking. Relay’s personal recognition layer uses calibration examples to identify useful measurements and distinguish rest, held poses, and short movements. Movement matching accounts for differences in timing between repeated performances.
Recognition is separate from activation. Confidence checks, deliberate holds, and release requirements help prevent an ordinary movement—or a gesture held for too long—from becoming repeated unwanted input.
For air handwriting, ONNX Runtime Web runs a pretrained character classifier locally. The swipe-word environment compares finger paths against a 20,000-word vocabulary using geometry and temporal matching.
Creator environments are validated configuration documents containing components, mappings, conditions, and selection rules. The same document can be previewed, installed in User space, exported, and remixed.
Independent workers keep inference separate from interface rendering. Camera video stays on the device, while published environment configurations and votes use a server-side database.
Challenges I ran into
Making personalization reach beyond gesture recognition. Recognizing someone’s movement is only useful if the layout, timing, feedback, and selection process also fit them. This pushed me to build environments that could vary independently of the underlying camera input.
Making Creator expressive without requiring programming. My first approach was too tied to specific environment types. I moved toward smaller building blocks: choices, groups, input mappings, targets, and selection rules. That made new combinations possible without implementing each interface as a separate feature.
Distinguishing recognition from intention. A detected gesture does not always mean “act now.” I had to account for rest, transitions, tracking loss, deliberate holds, and returning to a neutral position.
Following fast or imperfect movement. Hand tracking can lose observations during quick motion. I improved frame scheduling, tracking continuity, finger-counting geometry, and visual trails while keeping visual smoothing separate from recognition.
Making feedback useful. Persistent camera previews, current-gesture labels, hold progress, and a signal analyzer help users understand what Relay detects. These became essential tools for both operation and debugging.
Accomplishments I’m proud of
- Composing whole sentences using only eyelid movements in my demo. This showed that a small, repeatable signal could support original communication.
- Building a complete communication loop: calibrate, select, write, correct, undo, and speak.
- Supporting facial expressions, one- and two-hand interactions, held poses, and short movements within the same platform.
- Creating six different communication environments around one shared writing experience.
- Building a visual Creator that can assemble a functioning hand-controlled dial from reusable components.
- Connecting creation, installation, publication, remixing, and voting in a publicly accessible application.
- Passing 135 automated tests, asset-integrity checks, and browser checks covering key workflows.
Most of all, I am proud that Relay became something other people can create with.
I started by imagining interfaces myself. I ended with a platform that gives someone else the tools to contribute an idea based on experiences I may never have.
What I learned
The most important unit of personalization is the person, not the diagnosis.
I learned to think about observable abilities, comfortable movements, and individual preferences rather than assuming one interaction would suit a whole category of users.
I also learned that creating an interface and using it are different experiences. Both need to be approachable. A powerful recognition system is not enough if someone cannot configure it, understand its feedback, or recover from a mistake.
Separating reusable components changed the project. Once layout, movement, and selection became independent choices, the platform could support combinations that were not hardcoded as individual environments.
Finally, I learned that human creativity belongs inside the product. People with lived experience—and the people who know them well—can identify possibilities that I would miss working alone.
What’s next for Relay
My next priority is co-design and repeated-session testing with people who use alternative communication and the people who support them.
The eyelid-only demonstration establishes a working interaction on myself. Broader usefulness needs evidence across people, cameras, lighting, fatigue, and changing movement abilities. I want to measure setup effort, successful communication, correction rate, accidental activation, and comfort over time.
I also want to expand Creator’s supported inputs and building blocks, make calibration easier, and help people find environments that fit their needs.
The potential impact is personal: more control over everyday choices, more opportunities to express an original thought, and more ways to participate in a conversation.
The larger opportunity comes from sharing. An environment created for one person could become the starting point for someone else’s solution.
My vision is a platform where people can create the communication interface that works for them—and where the creativity of a community keeps expanding what is possible.
Built With
- cloudfare-workers
- computer-vision
- css3
- html5
- javascript
- machine-learning
- mediapipe
- node.js
- onnx
- python
- sqlite
