Almost every family has a visual impaired person. Their daily life quality is highly improved over the past years, due to technology capabilities. Although, some things are still not figured out. For example, visually impaired people can use social media like Facebook thanks to Digital Narrators, but what if they stumble upon a video? Wouldn't it be great to be able to analyze it somehow frame-by-frame and get an overall description of its content?

What it does

MyViewAssistant is a Skype Bot that helps people understand the content of a video by analyzing it frame-by-frame and sending back descriptions

How we built it

We used:

  • Microsoft Bot Framework and Azure and tailored the design for Skype.
  • Azure App Service to host the Bot code
  • WebJob to download and save the video
  • OpenCV to analyze and extract the scenes from the video
  • Azure Storage to store them
  • Computer Vision API to get the description of each frame
  • Language Understanding Intelligence Service for the User-Bot Interaction.

Challenges we ran into

  • A few difficulties binding all the different technologies and projects together
  • OpenCV requires the video to be stored locally but the Bot was running on Azure

Accomplishments that we're proud of

Despite the difficulties, we managed to connect all the projects and create a solution that is scalable on the cloud

What we learned

  • To start simple and add functionalities on the way
  • Work as a team
  • Explored new technologies

What's next for MyViewAssistant

  • Beta testing for Intelligence and flow improvement
  • Publish on Skype and other channels for public use

Built With

  • azure
  • azure-web-app
  • cognitive-services
  • computer-vision-api
  • luis
  • microsoft-bot-framework
  • opencv
  • signalr
Share this project: