• 

  Gesture and voice based controls activated, with hand tracking and cursor.

• 

  Adding a product to cart using gesture and voice based controls.

• 

  View of home page with ability to add products to cart.

• 

  View of cart items with ability to remove items from cart.

HoverFlow Navigation

By Brian Arthur and Jacob Laframboise

At Hack The North 2020 ++

A method of human-computer interaction (HCI) involving a combination of gesture-based navigation and voice commands to allow the user to explore an e-commerce site in an intuitive way.

Goal of this work

To demonstrate the potential of this system of HCI for an e-commerce site to allow users to interact with products with ease.

Libraries

We used Handsfree.js for 21-point hand tracking, see Handsfree Js for more information.

We used react-speech-recognition for the Web Speech API.

We used React and Material-UI to build our responsive sample e-commerce shop.

How to Use the App

Hosted with Heroku at HoverFlow

Navigate the e-commerce site using standard mouse controls or click the microphone icon button to open a new way to explore the shop.

Shop features include

• Add and remove items from the cart
• View cart items
• Responsive layout

The app asks for user permission to use the microphone and video webcamera.

Opens video feed tracking hand movements to interact with the app.

• Cursor will follow the position of the hand on the screen
• Pinch thumb and index finger to scroll the page

Creates a Speech Recognition stream to identify voice commands

• 'Activate' command to click element beneath motion cursor
• 'Go to home' command to go to the home page
• 'Go to cart' command to go to cart view

Built With

• handsfree.js
• html
• javascript
• react
• scss