What is CommandPlus?
CommandPlus is a Chrome extension that provides hand gesture-based control of one's browser. It does this by interacting with the user's webcam and sending video frames to a server for computer vision processing. If one of the predefined gestures is detected, CommandPlus triggers the relevant action in the browser. The extension is aimed at helping individuals who are limited in their movement or fine motor skills.
Inspiration
CommandPlus started with the aim of creating a more intuitive way to zoom in and out inside a browser window (hence, the name — Cmd+) to make it easier for elderly people as well as individuals with disabilities to read content online. Eventually, we also incorporated gestures to vertically scroll through the browser window without having to touch any input device. As Steve Jobs once said, there is no better input device that one's own finger(s).
Challenges & Accomplishments
Being our first time working on a Chrome extension as well as a computer vision project, all parts of the tech stack were pieces we had never worked with before. However, the most difficult part of the project was implementing the algorithm to detect the correct gestures from the videoframes using OpenCV (it didn't help that the entire room was painted in a skin-like beige colour :P).
What's next for CommandPlus?
We want to integrate more static as well as dynamic gestures to make browsing through the internet "hands-free" even more intuitive. Ideally, CommandPlus would have enough gestures in the near future to essentially emulate a virtual touch screen in the air using just one's webcam. At the same time, integration of advanced computer vision for eye tracking can also provide an additional level of convenience.
Log in or sign up for Devpost to join the conversation.