Shopping is Peanuts

MyCroft
Twilio

Inspiration

We wanted to develop a neighbourhood assistance service for people who have to stay at home. Elderly people often do not have the possibility to use online ordering services. Therefore, we have developed an automated telephone hotline and smart speaker app to place orders for groceries. Neighbours and other users can use a smartphone app to view the orders from people in their area and deliver it to them. Our solution builds upon existing telephone services for neighbourhood support assistants, but uses a fully-automated voice recognition and matching service which does not rely on manual processing of orders.

What it does

People who need groceries can place an order either by calling a telephone hotline or by using a smart speaker. The placed orders can be processed by other users who live nearby and are currently underway for shopping.

How we built it

We used different frameworks, such as the Mycroft framework and the Twilio API to enable the possibility to place orders either by calling a telephone (Twilio) or by a smartspeaker. The text of the placed order is sent via REST to the backend which calls a speech recognition and afterwards a named language understanding component (RASA) and manages the placed orders. The placed orders are sent to other nearby users and a virtual shopping cart shows up in their iOS application.

Challenges I ran into

Building such a tool requires overcoming some hurdles regarding the development and interplay of components, such as the Speech-to-text, natural language processing, the iOS application, the home automation framework and finally the backend. All these components have already shown that encapsulated they are able to solve problems for which they were built well. When one component parses wrong results or detections to another component it affects the whole framework.

Accomplishments that we’re proud of

The framework could help to bring generations in crises like the Covid-19 crisis together. We are also proud to have integrated a bunch of technologies while everyone in the team was working on tech pieces we’ve never used before: After being an Apple fan boy for years, Jochen finally experienced the greatness of swift development, Philipp developed his first MyCroft prototype two years after he told us about his first plans to do so, Cedric, our Rasa master, developed the NLU and Dominik learned that debugging a telephone-based voice app is not as funny as it sounds.

What we learned

All of us took the chance to learn new technologies. AI services such as NLUs (Rasa) or Smart Speaker APIs (Mycroft) Programming Apps with Swift Service to work with phone calls We also learned how to develop such a project with a remote team.