The Problem

Over 5 million Americans suffer from some sort of paralysis. Of those, around 60% are quadriplegic. A quadriplegic is someone who has paralysis of all four limbs. This is a life changing disability and it makes normal activities very difficult. When people lose access to their arms and legs, they don't have many ways to interact with the world. Often times, the best method is through speech. Phones and laptops today support many accessibility features that allow you to control the device with just your voice however we're still a long way from fully accommodating these limitations. One of the most famous quadriplegics is Stephen Hawking. He was a brilliant man and taught us many things. One area we are lacking in, is accessibility support for teachers and professionals with motor disabilities.

Slide presentations are ubiquitous in academic and professional life and while there has been work done on increasing accessibility for people listening to slide presentations, there has been very little done for the speakers. That is where we come in. We are SpeakEz - a voice-activated presentation assistant that solves these problems by leveraging natural language for slide transitions and dynamically generating content.

The Solution

Our vision is that you should have complete control of your slide presentation through your voice alone. To start, we support simple commands such as "next slide" or "previous slide" but we go much further than that. Your presentations don't necessarily need to have a specific order either, saying specific keywords can pull up slides in any order that you decide, regardless of the underlying structure of the slide presentation. Since we're making a smart presentation controller, we decided to make it even smarter. The presenter is able to dynamically pull in elements from the web during their presentation. If you're giving a talk about animals for example, you can say "consider the elephant" and an image of an elephant will appear in the deck even though it didn't appear originally. With these features we are making presenting a slide show more accessible than ever before.

Technical Innovations

SpeakEz is a web application that uses Node.js for the server and React for the frontend. We used the Spectacle JavaScript library to create in-browser slide decks that we could easily navigate through. Azure powers our core services and is the main driver of functionality in our app. We use Azure Cognitive Services' SpeechRecognizer class to perform our speech recognition. This natural speech is parsed and used for keyword detection during your presentation. The keywords use a combination of reserved trigger phrases (such as "next slide") and user-defined tags for each slide. To dynamically load elements from the web on the fly, we query the Bing Image and Entity Search APIs to obtain the description and suggested image for a particular keyword. We then generate the slide on the fly using a React modal.

Built With

Share this project: