Unlocked

Building conversation, community, and connection for those who need it most.

Comment

Inspiration

All of us have had close family members suffer from major strokes, leaving them with significantly reduced motor movement and control. Unable to move or speak properly meant that communication was never truly through themselves, stripping the connection from them and a world that unites through an online world, especially with family members scattered across the world.

Simple tasks such as hand-typing a WhatsApp message is a foreign concept due to the lack of fine motor control available in their hands. Through no fault of their own, they were isolated from those closest to them.

This is why we created Unlocked — the virtual keyboard and mouse system that utilises a facial scanning system to produce keyboard and mouse inputs, allowing someone to use a device without the need for hands.

What it does

The app takes in a live video feed through a computer webcam and tracks 468 facial landmarks that are used to produce distinct gestures, which are translated into unique keyboard and mouse inputs.

These gestures are user-specific and calibrated upon app launch, though we provide a recommended set of gestures that we believe results in the best ease of use.

Through these facial gestures, a user can navigate a keyboard through a D-Pad style selection, typing into anywhere possible such as a search bar, Word document, or chat. Users can also operate a mouse cursor through face tilting, activating the equivalent of mouse buttons through gestures.

How we built it

Unlocked is a real-time computer vision pipeline built from Python and other frameworks.

Your browser streams webcam frames to a server over a persistent WebSocket. There, Google MediaPipe extracts 468 facial landmarks and encodes them into an eleven-dimensional feature vector.

A k-NN classifier, trained on your own calibration data, evaluates that vector. Once a prediction clears our confidence threshold and holds steady across several frames, Unlocked fires a real OS-level keystroke directly into whatever application has focus.

Keyboard layout was designed with minimal effort in mind. Letters were arranged square in alphabetical order for clarity and optimised average distance travelled. Primary function keys such as Enter and Caps are placed in the centre for ease of access.

Challenges we ran into

We had initial calibration issues with respect to the facial gestures and how to distinguish them.

To overcome this, we initially underwent extensive physical testing to see which facial gestures had a higher level of distinction. We also played around with the tolerances, testing how sensitive the model was to movement and at what point it would consider a facial movement an actual intended gesture meant to produce an input.

The final change we actioned was to increase the calibration time for each input/facial gesture, allowing the model to have an increased understanding for how a user might, for example, raise their eyebrows or smile for slightly longer time periods with variation.

As the intended use is limited to facial gestures, we split the app into two states: keyboard and mouse. There were issues in how to separate/toggle between these states seamlessly.

Another significant issue was with the cursor. After an initial head tilt to move the cursor, if the user returned to the 'centre' position, the model's 'centre' would drift, causing the user's neutral head position to register with some degree of tilt, resulting in unwanted cursor movement.

The first fix was to anchor this tilt reference point to the user's chin. This improved the problem; however, there was still a small amount of drift which prevented any accurate mouse clicking from happening. By calibrating the centre position and actively centring the chin position...

Accomplishments that we're proud of

Getting the initial face tracking working with a live visual feed was a great motivator at the start of the hackathon. This was important and something we were proud of, as it is the foundation of the system, and it looks cool...

This also happens to be our first project together, and we are proud of the chemistry and synergy we fostered in only 24 hours. We were also pleasantly surprised to find a topic that resonated with all of us through our own personal connections.

What we learned

Collaboration in a high-pressure environment is all about communication and understanding your own strengths and weaknesses to optimise team efficiency and work quality.

As we are a team of friends, we understood this and delegated tasks based on this.

What's next for Unlocked

The primary future step for Unlocked is to distribute it to real users. Partnering with rehabilitation centres, doctors, and therapists will greatly help with getting the app out to those who need it. Only then will we see the real effects of Unlocked.

We will look to creating per-user accounts to ensure smooth expansion in the future. We will look more deeply into database management systems and privacy management.

Multi-server deployment will help us connect users all across the globe and enhance connection for those who need it most.

Also accounting for variable conditions eg. lighting, skin tone, face shape etc. will be a big step in increasing accuracy across a wide range of users

Built With

Share this project:

Updates