I have tried different symptoms checker mobile apps, but current there is none that could automatically identify the human features and provide possible symptoms from the human image picture itself. What if we leverage the power of Azure Cognitive Service and Azure Search to extract human features (face landmarks, gender and etc...) to quickly provide symptoms and learn about our conditions in a matter of seconds.

What it does

SympMD is your daily mobile health symptoms checker. By just taking a selfie, the app will able to provide all possible symptoms based on the generated age and gender from azure cognitive computer vision service. From the selected symptoms, it will able to provide you all possible health conditions that may you currently experience. Also, it will able to provide you all the know-hows about your selected health condition. Also, it has a "Discover Alternative Medicine" feature wherein you could discover different medicinal usage of some plants/herbs by just uploading a picture of it.

How I built it

For the Mobile App:

  • SympMD mobile app was built using Xamarin Forms. For the purpose of this hackathon/demo, you can only use the android version and you can try it here:

For the Back-end Service:

  • Azure Function App was used to act as the API service of the mobile application, it is responsible to secure different the API keys and fetch data from different azure cognitive services.
  • Azure Cognitive Computer Vision Service was used to identify human features (face landmarks, gender and etc...).
  • Azure Search was used to index and search all possible symptoms and health conditions data. All the data were gathered from different royalty free sources (books and websites).
  • Azure Cognitive Search Service (Bing Web Search) was used to get the different health condition know-hows and alternative medicine medicinal usage.
  • Azure Cognitive Search Service (Bing Image Search) was used to collect herbs/plants image and used it to training of the model in Custom Vision AI.
  • Azure Cognitive Custom Vision Service was used to create and train the model of the plants/herbs images and identify plants/herbs images.

Challenges I ran into

  • It is hard to find royalty free sources about health symptoms and conditions.

Accomplishments that I'm proud of

  • I was able to used different azure cognitive services and integrate it in a real world application.
  • I was able to utilized the power of azure search and leveraging the index attributes (Retrievable, Filterable, Sortable, Face-table, and Searchable)
  • I was able to built a mobile application that uses azure search and azure cognitive vision services.
  • I was able to collect plants/herbs image using Bing Image Search and used it to train the model in Custom Vision AI.

What I learned

  • I have discovered the different azure cognitive services and their usage.
  • I have learned how fast to import data and create an index in azure search.

What's next for SympMD

SympMD will be publish on Google Play Store and App Store in the next few months after all the stabilization has been made for its MVP version. SympMD is currently on a alpha phase, any suggestions and comments from the Devpost community and medical community will be appropriated.

Built With

+ 10 more
Share this project: