Full Application View (Doctor left, Patient Right)
Bi-directional channel based video communication implemented with webRTC and coupled with Azure Cognitive Services for facial analysis.
Automatic speech-to-text natural language processing for a seamless chat experience with a latency of <500ms.
Dynamically generated dialogue summary with intelligent text-analytics for sentiment and medical key-phrase highlighting.
TeleHealth is currently being deployed in hospitals and care locations to improve access, fill gaps in care, provide 24/7 services, and adapt to demand for access to medical specialists. To build on the growing technology of TeleHealth, we wanted to introduce both cutting-edge machine learning applications as well as modern security practices to the field.
What it does
In the shift to demand-driven health care, TeleHealth is becoming the patient’s first — and most frequent — point of access for urgent care, triage for emergency conditions, specialty consults, post-discharge management, chronic care management and more. Health on a Cloud is designed to streamline the interaction between the patient and the healthcare provider, while leveraging Microsoft Azure's numerous cognitive services to enable a model of care that is ubiquitous and seamless. Health of a Cloud acts in 3 ways: Patients can have a personalized, anonymous, one-on-one interaction with the healthcare specialist they need; Administrators can manage multiple TeleHealth sessions in reliable and secure access rooms; and Doctors have access to a powerful, tailored AI dashboard to provide the care that their patients deserve.
How we built it
We used React.js to build the front-end of our webapp, and we built our backend api using Express.js, which is deployed using Heroku. We used tokbox and webRTC to build our Audio/Video session manager, and websockets to drive our live chat streams. We took full advantage of Microsoft Azure's cognitive services, using FaceAPI for facial recognition and analysis, Text Analysis API for natural language processing and sentiment analysis, Azure Speech Services for speech to text purposes, and more for interfacing and synchronizing purposes.
Challenges we ran into
Another technical challenge that we faced was that we had a difficult time fully implementing the ideology of anonymity that we truly believed in. We wanted the users of our application to feel innately comfortable with our services, without ever having to be afraid of having their data stolen or used for other purposes. After hours of trial and error, we were finally able to over come this issue by implementing web sockets for anonymous user-chat functionality, and fully utilized Heroku cloud deployment to anonymously send API requests over the web, without any unwanted caching or data retainment.
Accomplishments that we're proud of
We were extremely happy and proud of the fact that with the power of Azure Cognitive services, we were able to reach across disciplines and apply machine learning to fit the needs to remote healthcare providers without having to breach our patient's privacy or put their data at risk of breach. Ultimately, we were able to implement a secure and scalable application architecture while being able to implement a TeleHealth care system that is designed for seamless patient-provider communication.
What we learned
This was our first time working with Azure's cognitive services API, and we learned a lot about making asynchronous requests with potentially expensive queries or resource hungry operations to achieve minimum latency in our video and chat apps client-side. We also learned about the struggles that end-users potentially have with TeleHealth services, and why they face these struggles. We used these philosophies as the driving factor of our project prototyping and design, ultimately allowing us to actualize our final product. Finally, we learned how bi-directional live communication can be achieved through leveraging WebRTC and WebSockets for truly live (latency < 500ms) and interactive communication.
What's next for Health on a Cloud
For the future, we are looking to implement even more of Azure's cognitive services portfolio, such as their Ink Recognizer, as a potentially a powerful tool for healthcare providers rooted in pen and paper administrative systems to break out into a digital platform, or Azure's computer vision services to identify hazards or anomalies during a TeleHealth session and ensure the safety of both parties. We also want to look into integrating a secure sign in and authentication system using OAuth and Firebase, in order to fully realize our application as the first TeleHealth effort focused on privacy and security.