I was inspired to make this hack after see how many young and old people alike that have physical and/or visual impairments struggle to use their computers to their full potential because of how challenging it can be to use a browser since that requires fine motor controls for navigation and ability to see all the small details on a page.
What it does
It uses a Google Home as a voice interface to browse the web through an accompanying Chrome Extension that manipulates Google Chrome. Its current features include: go forward/back, invert colors, mute/unmute tab, new tab, open link, reload page, restore window, scroll down/up, close tab, site search, pause/play youtube video, and zoom in/out.
How I built it
I made a Google Cloud Function that runs server-side logic for processing user requests through the Home. One of the things the GCF does is it updates Firebase values for a user so that the Chrome Extension can pick up on the updates and change Chrome's state accordingly. Also, Dialogflow is used for the NLP aspect of understanding user requests.
Challenges I ran into
- syncing code folder to GCF bucket
- finding a way to synchronize actions/values between Action and Extension
- authenticating/getting Google user on both platforms (OAuth 2)
- figuring out what commands would be most useful for the end-users
- getting the Google Home connected to the internet because the password-protected WiFi was making this difficult
Accomplishments that I'm proud of
Overcoming the aforementioned challenges! This was a non-trivial project to make work considering the various parts needed to be built.
What I learned
Some of the things I learned include: how to use Firebase in a creative way, how OAuth2 works, how to make a Chrome Extension, and how various Chrome APIs work.
What's next for Chrome Control
- More features: There are so many more features that can be added. As long as there's a Chrome API for manipulating some aspect of Chrome, I believe I can find a way to make it accessible through voice.
- Deploying to other common voice platforms such as Alexa and Cortana to increase potential user base.
- Also adding the totally different feature of eye-tracking based navigation for scenarios where voice-based navigation may not be preferable.