I was inspired to make this hack after see how many young and old people alike that have physical and/or visual impairments struggle to use their computers to their full potential because of how challenging it can be to use a browser since that requires fine motor controls for navigation and ability to see all the small details on a page.

What it does

It uses a Google Home as a voice interface to browse the web through an accompanying Chrome Extension that manipulates Google Chrome. Its current features include: go forward/back, invert colors, mute/unmute tab, new tab, open link, reload page, restore window, scroll down/up, close tab, site search, pause/play youtube video, and zoom in/out.

How I built it

I made a Google Cloud Function that runs server-side logic for processing user requests through the Home. One of the things the GCF does is it updates Firebase values for a user so that the Chrome Extension can pick up on the updates and change Chrome's state accordingly. Also, Dialogflow is used for the NLP aspect of understanding user requests.

Challenges I ran into

  • syncing code folder to GCF bucket
  • finding a way to synchronize actions/values between Action and Extension
  • authenticating/getting Google user on both platforms (OAuth 2)
  • figuring out what commands would be most useful for the end-users
  • getting the Google Home connected to the internet because the password-protected WiFi was making this difficult

Accomplishments that I'm proud of

Overcoming the aforementioned challenges! This was a non-trivial project to make work considering the various parts needed to be built.

What I learned

Some of the things I learned include: how to use Firebase in a creative way, how OAuth2 works, how to make a Chrome Extension, and how various Chrome APIs work.

What's next for Chrome Control

  • More features: There are so many more features that can be added. As long as there's a Chrome API for manipulating some aspect of Chrome, I believe I can find a way to make it accessible through voice.
  • Deploying to other common voice platforms such as Alexa and Cortana to increase potential user base.
  • Also adding the totally different feature of eye-tracking based navigation for scenarios where voice-based navigation may not be preferable.

Built With

Share this project: