Health on a Cloud

Full Application View (Doctor left, Patient Right)
Bi-directional channel based video communication implemented with webRTC and coupled with Azure Cognitive Services for facial analysis.
Automatic speech-to-text natural language processing for a seamless chat experience with a latency of <500ms.
Dynamically generated dialogue summary with intelligent text-analytics for sentiment and medical key-phrase highlighting.

Inspiration

TeleHealth is currently being deployed in hospitals and care locations to improve access, fill gaps in care, provide 24/7 services, and adapt to demand for access to medical specialists. To build on the growing technology of TeleHealth, we wanted to introduce both cutting-edge machine learning applications as well as modern security practices to the field.

What it does

In the shift to demand-driven health care, TeleHealth is becoming the patient’s first — and most frequent — point of access for urgent care, triage for emergency conditions, specialty consults, post-discharge management, chronic care management and more. Health on a Cloud is designed to streamline the interaction between the patient and the healthcare provider, while leveraging Microsoft Azure's numerous cognitive services to enable a model of care that is ubiquitous and seamless. Health of a Cloud acts in 3 ways: Patients can have a personalized, anonymous, one-on-one interaction with the healthcare specialist they need; Administrators can manage multiple TeleHealth sessions in reliable and secure access rooms; and Doctors have access to a powerful, tailored AI dashboard to provide the care that their patients deserve.

How we built it

We used React.js to build the front-end of our webapp, and we built our backend api using Express.js, which is deployed using Heroku. We used tokbox and webRTC to build our Audio/Video session manager, and websockets to drive our live chat streams. We took full advantage of Microsoft Azure's cognitive services, using FaceAPI for facial recognition and analysis, Text Analysis API for natural language processing and sentiment analysis, Azure Speech Services for speech to text purposes, and more for interfacing and synchronizing purposes.

Challenges we ran into

The implementation of state handling and life-cycle management in our React components had a very hard time synchronizing with multiple API requests simultaneously. We were able to fix this by taking full advantage of the asynchronous nature of javascript, as well as the conditional rendering abilities of React to properly display the appropriate layout.

Another technical challenge that we faced was that we had a difficult time fully implementing the ideology of anonymity that we truly believed in. We wanted the users of our application to feel innately comfortable with our services, without ever having to be afraid of having their data stolen or used for other purposes. After hours of trial and error, we were finally able to over come this issue by implementing web sockets for anonymous user-chat functionality, and fully utilized Heroku cloud deployment to anonymously send API requests over the web, without any unwanted caching or data retainment.

Accomplishments that we're proud of

We were extremely happy and proud of the fact that with the power of Azure Cognitive services, we were able to reach across disciplines and apply machine learning to fit the needs to remote healthcare providers without having to breach our patient's privacy or put their data at risk of breach. Ultimately, we were able to implement a secure and scalable application architecture while being able to implement a TeleHealth care system that is designed for seamless patient-provider communication.

What we learned

This was our first time working with Azure's cognitive services API, and we learned a lot about making asynchronous requests with potentially expensive queries or resource hungry operations to achieve minimum latency in our video and chat apps client-side. We also learned about the struggles that end-users potentially have with TeleHealth services, and why they face these struggles. We used these philosophies as the driving factor of our project prototyping and design, ultimately allowing us to actualize our final product. Finally, we learned how bi-directional live communication can be achieved through leveraging WebRTC and WebSockets for truly live (latency < 500ms) and interactive communication.

What's next for Health on a Cloud

For the future, we are looking to implement even more of Azure's cognitive services portfolio, such as their Ink Recognizer, as a potentially a powerful tool for healthcare providers rooted in pen and paper administrative systems to break out into a digital platform, or Azure's computer vision services to identify hazards or anomalies during a TeleHealth session and ensure the safety of both parties. We also want to look into integrating a secure sign in and authentication system using OAuth and Firebase, in order to fully realize our application as the first TeleHealth effort focused on privacy and security.

Built With

Submitted to

Hack the North 2019

Created by

I worked on building out the front-end react app on the client-side, and integrated computer vision and facial recognition into the application using Azure Cognitive Services. I implemented the video call feature with TokBox and WebRTC.

Bill Liu
Ex-SWE Intern @Facebook, CS @UCLA
I worked on deploying our backend to heroku and establishing the sockets communications between our client and the backend. On the client-side, I wrote half of the react front end and set up azure speech to text.

Frank Wu
I worked on the back-end to implement bi-directional communication with socket.io. On the client-side, I used Azure Cognitive Services to integrate natural language processing and facial analysis features into the application.

Kevin Li
Yiming Tan