We provide a scalable way for people to receive recommendations for action by telephone. Besides addressing specific questions about symptoms, we also assess the phonetics of the voice and can thus identify not only the semantic context but also dysphonia. Dysphonia is a reliable indicator of shortness of breath, which is an acute symptom of COVID19 disease.
Using AI-based real-time processing, we also detect coughing and other audible symptoms and can, therefore, provide accurate diagnostic recommendations for hotline agents. This 'hidden' support should be beneficial for agents, as users often tend to report false symptoms to increase the likelihood of being tested.
We respect all privacy policies in this regard by avoiding the storage of any personal data. To be able to continue to utilize the given information, we encode the entire conversation into a word pair, which the caller receives at the end of the conversation. With the help of these word pairs, doctors/clinics and research institutions can reconstruct the data and link it to the corresponding person, as long as the patient shares the word pair.
In collaboration with the field specialists Prof. Hugo Alexandre Ferreira from the University of Lisbon, Dr. Jesus Flores Soler from the Universidad de Madrid, and previously with Dr. Alexander Thieme from the Charité Berlin, we have broadened our initial release with the functionality of AI-controlled symptom recognition.
We currently have automated hotlines translated into three languages. English, German and Danish. A German version has already been handed over to Charité Berlin for pilot testing and we are part of a think tank at Charité to develop self-triage solutions. Unfortunately, the hotlines reflect different development stages. We recommend the English hotline because there we are most advanced with Natural Language Processing.
Just call +1 833-700-1779 and visit our landingpage. These automated hotlines do not have to be automated. They can also be used in a supportive way. The superior advantage of our hotline is that it is possible to take the anamnesis which is done over the phone with you.
An average patient is interviewed on average four times before being tested. We want to reduce this process to one time.
Every hospital working with SAP can already use our system to import the COVID medical history directly into the hospital information system.
Recognition of Symptoms Over the Telephone
We, as a provider of a self-triage solution, we often have to deal with the problem that users intentionally or unintentionally provide false information. Therefore we took a look at the most critical symptoms that play a decisive role in a COVID diagnosis.
This diagram shows the qualitative behavior of symptoms of COVID-19 vs cold and influenza based on COVID-19 and Computer Audition: An overview on what speech & sound analysis could contribute to the SARS-CoV-2 Corona crisis: https://arxiv.org/pdf/2003.11117.pdf
As you can see, the essential evaluation factors for COVID-19 can be detected by a change in breathing patterns and dysphonia. Hence, algorithms should be able to evaluate dysphonia caused by COVID19 over a telephone line. As a result, it should not be possible to pronounce Xen & Xan (EPA) Sounds (e.g., Spanish: Hijo) the right way while having dyspnea. Read more about how dysphonia and breathing problems correlate here
Guide of the Madrid Health Department, with nonverbal assessments
To some up, soft skills are required to query the symptoms from the patient. Moreover, it unfortunately implies that this query is very subjective. We want to intervene in the process in a supportive manner and objectively identify symptoms.
Technical Solution of Symptom Extraction
To provide the hotline, Amazon Connect is used as a scalable customer center solution. On this basis, a chatbot was programmed, which interacts with the user via voice input. Initially, Amazon Lex and AWS Lambda are used for speech recognition and validation.
In this hackathon, we have increasingly relied on public data, which we downscaled to the same level of quality, an ordinary telephone line, with the help of equalizers and compression algorithms. This working method does work, but with a few weaknesses. Therefore we ask patients to repeat a sentence to build a reasonable data structure.
We use the recordings of the phone calls and send them to the Amazon Transcribe instance. By extracting the identified words, we can evaluate clear gaps between spoken words and breath sounds and use these as indicators for determining the probability of COVID19 disease.
As a basic setup, we use the CloudFormation from AWS Media Analysis. Our previously developed pipeline for recording conversations was connected to this architecture by a helper Lambda function. (https://aws.amazon.com/de/solutions/implementations/media-analysis-solution/)
For testing, we initially used free sample data, but we soon realized that we needed real data for our case. So we made telephone calls with C+ patients who were pronouncing sentences we had defined. Unfortunately, the number of sentences is still too small to be able to make an accurate statement about dyspnea.
used samples can be found here:
If you want more details, about the structure, scroll down to the old part to get an overall look.
Sharing Your Answers With a Simple Word Pair
To encode them in a user-friendly way, your answer and diagnosis combination are mapped to a word pair from the dictionary. The customer has verbally conveyed the word pair together with the recommendation after the survey. He can memorize it or note this word pair and share it with a doctor. A further service developed with Ionic and provided as an app can be used by health care personnel to generate and view the corresponding medical history from the word pair. Furthermore, a QR code is also created for which a procedure for integration into the hospital information system already exists.
Every hospital that works with SAP can already use our system to transfer the Covid medical history, which has already been obtained on the phone, directly into their system.
It is important to us to realize our solutions under consideration of data protection. Therefore we do not want to collect personalized data in the normal hotline and discard all the data after the call.
However, we need a basis for the algorithm, for this we offer a second hotline, whose only purpose is a voluntary audio donation. We want to ask diagnosed cases to call the second hotline and repeat three short sentences. On top the patients need to answer four questions about symptoms.
Cov2Words is an automatic hotline developed for COVID19 anamnesis of risk groups. Having started at the #WirVsVirus, we leveraged our solution in the #EUvsVirus. The basis relies on the anamnesis questionnaires provided by virologists at the Berlin Charite. This approach has been refined and iterated in the past weeks. The feedback from mentors and stakeholders opened numerous partnerships and ideations all over Europe.
Our hotline is currently in the testing phase at the Charite Berlin. By winning the “Pandemic Response” hackathon of the US government, we obtained further insights into the requirements and increased our reach to collect more feedback from experts.
Our primary focus over the past hackathons was to listen carefully for the pains of stakeholders and develop a solution, such as multi-language support, the flexibility to configure different questionnaires, or interfaces to hospital information systems. Therefore, we tried to iteratively refine our solution by brainstorming with experts and identify their demands.
During the #EUvsVirus, Prof. Ferreira and Dr. Soler joined Cov2Words for providing new impulses, that have empowered our solution from a chat-bot-based hotline to a real-time dysphonia analysis by using machine learning. This increment adds clear value to our solution and defines a technological USP.
Hence, our strategy is to professionalize our prototype into a reliable technology that is easy to use and ready for cost-efficient scaling. On the one hand, we plan iterative testing to reach the quality criteria required for a production rollout. On the other hand, we search partners to reach as many stakeholders in the health sector as possible. Moreover, we see the need in different application scenarios among other sectors as well.
We are aware that the rollout step is probably the most difficult. Nevertheless, we are pushing Cov2Words towards clinical use to control the spread of COVID-19 and involve people of all generations and social layers.
In recent weeks, we have seen fantastic work in these areas in other hackathons and scientific studies. We are now trying to collect these findings and integrate them into our already working hotline solution.
Together with the leading experts in the virology of the Charité Berlin, we made a chatbot based hotline available, which advises you if you should get tested or not. This weekend we would like to make the hotline ready for deployment in German and translate it into English.
just call +1 877-201-9777 from the US or +372 668 2808 from Swiss / Germany or +45 80 82 01 45 from Denmark and visit our landingpage Both in Europe and the USA, the hotlines supports all 29 questions, and the first test pilots started last week in Berlin.
How testing works in Germany
Germany is one of the leading countries in testing on COVID-19, with probably one of the lowest dark figures worldwide. This is mainly due to a skillful pre-selection of who is invited to a test. The patients are pre-selected based on 29 questions and divided into categories. The categories can be simplified with:
everything is fine but stay at home monitor your symptoms go into self quarantine go get tested
Which recommendation the patient receives is based on his symptoms, his exposure to other patients who have tested positive or his recent stay in risk areas. Besides, exclusion procedures are applied to see if, for example, it is only flu.
This questionnaire is currently filled by assistants on the phone and then entered manually into a web application. The used app https://covapp.charite.de/ is already provided in English by the Charité Berlin.
CovApp - Questionnaire by Charité
Within the web application CovApp the patient needs to answer 29 questions. These questions try to inquire the medical history of the user:
What are his symptoms Has the user had contact with a positive tested or was in a risk area is the user part of a risk group
Following up on the questions, the answers are evaluated with the help of a decision tree in which each hospital can define themself, and a recommendation is given on how to behave. In addition, the user receives a QR Code - in which his answers are encoded as XML. This QR code is used to transfer the data to the hospital information systems.
Our approach - chatbot-based hotline using AWS
To provide the hotline, Amazon Connect is used as a scalable customer center solution. On this basis, a chatbot was programmed, which interacts with the user via voice or text input. Amazon Lex and AWS Lambda are used for speech recognition and validation. The current goal is to provide an interface where each hospital can create its decision tree. However, this interface will probably not be realized by us. In addition to our hotline solution, there is already an app and a webpage solution, which will be implemented this week. We only share some medical history based on 30 chatbot question, the hospital interprets it and can give their recommendation.
Simple word pairs - to be used as analog QR-code
The response possibilities amount to approximately 300 million combinations. To encode them in a user-friendly way, each combination is mapped to a word pair from the dictionary. The customer has verbally conveyed the word pair together with the recommendation after the survey. He can memorize it or note this word pair and share it with a doctor. A further service developed with Ionic and provided as an app can be used by health care personnel to generate and view the corresponding medical history from the word pair. Furthermore, a QR code is also created for which a procedure for integration into the hospital information system already exists.
Work in The "Danish Weekend"
Our challenge for the Danish weekend was to come up with a solution that would allow IT inexperienced people, to build their hotline. In the previous weekends, we have already developed an English and a German solution. To have a usable product, we had to create an interface that allows doctors to submit questionnaires themselves.
How we solved the problem
It was essential to consider the Amazon Connect structure in the background and, at the same time, provide an evaluation at the end of the process. On the other hand, the tool should not become too complex and incomprehensible. Unfortunately, the task of developing such a tool is very technical and time-consuming. The result might look relatively unspectacular, but is still the most ambitious part we have implemented yet. We had to adapt our hardcoded solutions, which we have been working on over the last two weeks so that it can handle flexible inputs.
This figure illustrated questions input fields.
You can test it under Github.It still needs some UX-Design, but works so far.
Technical Side Danish problem
For the solution, we used a react redux framework, which creates JSON files at the end, which are then loaded back into amazon connect. In the Amazon instance, contact flows are created from the Json file, which define how the conversation will take place. In addition to the contact flows, the parameters for the evaluation Json are also set.
You can see it on this
This figure shows realized use cases depending on frontend, backend and amazon connect instance.
This flow shows the procedure of the solution. To make it easier to understand, it is a simplified variant that does not represent all processes on the server-side.
This diagram is more complex and shows how the web app works in detail.
We are trying to implement a client-side solution for the Charite in the next days, and we are in contact with the responsible persons of the University Hospital. At the moment, our solution only works in hospitals/practices that use SAP. We want to use a medical standard and then connect it to the various information systems. We want to offer our service and provide a medical consultation via phone to another population group!
Within the #WirVsVirus Hackathon, we as a team consisting of Aaron Szerencses, Alexander Schoenhals, Anna Lueders, Christopher Seidemann, Fabian Lueders, Philip Ehret, Ruben Koch, and Thomas Hepp developed a solution in exchange with the responsible persons of CovApp!
Many thanks to the great Hackathon mentors for the support and help. Also, we would like to thank Amazon for providing the server structure. Many thanks to the Charité, who gave us insights into the structure of their app and actively supported us.
Many thanks to the team of the Danish Hackathon. The mentors who supported us were great and the choice of mentors, even better. It's crazy that you managed to get us together not only with a hotline operator, but also with someone who programmed hotlines.
Unfortunately, we had so many meetings over the course of the weekend that we couldn't take full advantage of this support. But thanks a lot to our mentors and the organizers for all their effort.