K-64-HajjAssistant

Smart headphones with Hajj Assistant that acts as a personal guide to the pilgrims.

Comment

Inspiration

Every year, pilgrims face many problems in the communication and guidance, more detailed:

  • People experiencing pilgrimage for the first time, exhibiting lack of knowledge about different aspects of Hajj.
  • The wide variety of Hajj rituals details that are usually hard to remember.
  • The urging need of personal guides throughout the Hajj to acquire different information about different places.
  • Many of the pilgrims cannot read the guidelines due to language differences or their inability to read.

What it does

According to "The Economist", few numbers of pilgrims have smartphones, so, a mobile application solution may not be suitable for many people that can't deal with technology. So, we need a simpler way that's easy to use and doesn't require any expertise. For our solution:

  • It's a wearable smart headphone device with a GPS module that takes speech inquiries through an embedded microphone. It recognizes the content of the query and responding with the relevant reply; this includes information about the current location, useful information about the holy texts and transcripts related to the sacred places the pilgrim visits.
  • It tracks the user's Hajj journey step by step to provide information about what's next.

How we built it

For the Assistant we used:

  • DialogFlow: the working brain that represents the intelligent part that provides the user with all the information.
  • Web Speach API: to perform the speech-to-text and text-to-speech conversion, as DialogFlow takes an input text and outputs a text.

  • Nodejs (Express.js, socketIo.js): to build the web application demo.

  • We used a modified version of an opensource repository on Github for the Voice User Interface (VUI) to demonstrate the prototype, and we combined it with our Hajj Assistant. https://github.com/girliemac/web-speech-ai

Challenges we ran into

  • Unavailability of the hardware materials for the device to build the actual product. So, we simulated the process using a web application as a proof of concept.

  • The device's core is mainly based on Natural Language Processing (NLP), for which the "Arabic" language technical support, in terms of frameworks, APIs, and libraries, is limited. Besides, many Arabic dialects make the task difficult to tackle in a narrow time window. So, we only considered the English language as a kickstart.

Accomplishments that we're proud of

Day_1: We combined a Voice User Interface (VUI) with our Hajj Assistant. We tested it for different situations and analyzed its responses.

Day_2: We empowered it with the "GPS", to properly tell the location. We're working on connecting the assistant to firebase database to get the needed information.

What we learned

We learned how the assistants work; how they know intents, respond to the user, send information and following up the user in his actions.

We learned how the DialogFlow API works and learned how to connect it to our web simulating app.

What's next for K-64-HajjAssistant

There are some advanced Challenges and Future Work to be considered:

1- Multilingualism: As a proof of concept, we only supported the English language. So, we're planning to support multiple languages to meet the needs of pilgrims from different countries.

2- Cocktail party problem/Blind Source Separation: as many people are crowded, noise can affect the device ability to properly detect what the user is saying. We can solve this problem by using Blind Speech Separation (BSS) technique based on "Independent Component Analysis" (ICA) that acts like a selective attention while filtering out a range of other sounds to separate blindly mixed recorded signals even from only one stream of noisy speech is available.

3- Network Connection: In the large crowd of pilgrims, many devices are competing for the attention of a cellular network's antennas. Networks are further strained because these are times when people are apt to use data-heavy applications, which are all reasons to weaken or even stop the device's connection. This can be solved by regularly storing relevant data based on location and buffer them temporarily in order to provide an offline service.

4- Embedding a heat sensor to monitor the temperature to notify the user when he needs to cool up.

Share this project:

Updates