The planned domain for the project is but currently we are still working on getting the Facebook message to integrate with this website. The integrated messenger is visible on the test server but unfortunately it is not publicised by Facebook yet as we are waiting on their review so it is only usable by our accounts.


We were inspired by the Mediworx challenge to make faster healthcare in Slovakia. One of the best ways of doing this seemed to be lightening the load on the healthcare system by helping people perform basic diagnosis at home.

What it does

Florence is a medical chat bot that can help users diagnose symptoms from home. Users use the interface of Facebook messenger or a simple web app which is very accessible the user can diagnose symptoms and also submit images of skin spots to do automated skin desease detection using machine learning. It's also possible to upload a recording of a cough and classify it as safe or a specific disease again using machine learning.

How we built it

The whole application is running on a google cloud server. This has several flask servers which serve the front-end web page with the integrated messenger and also run the bot that handles the automated messaging. The information for the medical diagnosis is scraped from the UK NHS 111 website for non-emergency conditions. Two machine learning models (skin disease detection and cough classification) use different machine learning techniques. The melanoma detection uses a google cloud platform convolutional model which is trained on a custom dataset and the cough detection machine learning was done with an LSTM recurrent network trained from scratch on spectral features of each second of the recordings in the dataset.

Challenges we ran into

The biggest challange was making the chatbot. We struggled with interfacing with Messenger and the NHS website got concerned about our health and nearly sent an ambulance. Making the cough analyzing RNN was also quite challenging since none of us understand sound spectral features and there was little data available.

Accomplishments that we're proud of

Having a beutiful front end and getting machine learning models to work.

What we learned

We learned to make a model using spectral features of an audio track. We also did our first transfer learning on google cloud. Doing the chatbot we gained a lot of experience with interacting with Facebook (through messenger API) and NHS there.

What's next for Florence

Many other diseases can be diagnosed by machine learning from simple photos - for example mucus and urine color, eye redness and back assymetry can be indicative of various diseases. The audio analysis also needs to be retrained on a larger range of data. For all these further projects we would neet to get data directly from the helthcare providers.

Share this project: