Point Right

Inspiration

Asking someone to change slides or having to click a space bar during a presentation is annoying. And what if you forgot your clicker?

What it does

Our Chrome extension "Point Right" lets you change slides in Figma without having to click a thing. Using Google's MediaPipe framework, we mapped finger/hand gestures to actions in Figma slides, where pointing two fingers goes forward and three back.

How we built it

Our code has three main components. Our panel has a simple UI that lets you configure the extension. It is also the "brain" that connects our engine to the bridge. The engine uses MediaPipe's gesture recognizing tools (the AI library trained to identify the joints in the hands) to trigger actions by our gestures.js file. The bridge is what connects our engine to Figma through the extension.

Challenges we ran into

At first, we wanted one finger to move forward and two to go back, but during normal hand movement as you talk, actions were triggered when unwanted. So we changed to two and three fingers.
MediaPipe has a built-in recognition of a two finger gesture but it only worked if you held them up right, which isn't very natural during a presentation and it didn't have a three finger option. So, we mapped the distance between fingertip and wrist to decide how many fingers were held up and to be able to hold your hand facing down.
Finally, actions still triggered at random moments, but we found that requiring to hold the gesture for 8 frames instead of the default 4 significantly reduces the amount of accidental actions.

Accomplishments that we're proud of

We're proud of how we organized our workflow to get this done surprisingly quickly. We divided tasks, brainstormed and communicated throughout without walking over each other. We also learned a lot about JavaScript! We had all touched the code before but we are actually network and security students who really enjoy tinkering. We had an idea, had all explored MediaPipe and we were able to use the tools at our disposal to bring our idea to life. We also didn't go over the top, we made sure that what we had not only worked but we also understand what we did.

What we learned

We learned that being more precise is favorable in this type of code and that testing every scenario possible is key! Sometimes it works when you try it but when someone else does it different it breaks. As mentioned above, we learned a lot about JavaScript and how to use it to connect different platforms as it goes beyond our pre-existing knowledge of the tool.

What's next for Point Right

More gestures and actions, for example, in a Figma Design you could move around and zoom in with your hands. We could also integrate this to a tool outside of the browser so you don't necessarily have to use your webcam.