DiagnoSEE

Homepage
Dashboard
Comparison between original video and GAN generated video.
Logo
Comparison of original video frame (left) and high-resolution video frame generated from the website (right)

Inspiration

My cousin experienced an overbite for his braces during quarantine. While this would be a small problem during non-COVID times, I saw first hand the inefficiencies of telemedicine through Zoom. During the initial session at MedHacks, my team member learned from experts that low video quality is a major barrier for high-quality care in telemedicine. Thus, we created a deep learning solution that allows physicians to access high-res videos for their telemedicine appointments while lessening the technological strain on patients.

What it does

The website uses generative adversarial networks (GAN) to create high quality videos from lower resolution patient videos. This is especially useful for physicians that need to perform a physical examination, who would otherwise struggle without high-def visual interaction (Dermatologists, pediatricians, geriatricians). Patients can upload videos from their telemedicine examinations and select one of the many physicians available to give comments on his/her condition. Our server processes uploaded videos and generates the higher-resolution version of the video automatically. Patients can then go on to their dashboard, which provides a link to view the high resolution video alongside the original video, enter any notes, and submit the videos to the physician. On the other end, physicians can login to the website to view all submitted videos from patients and provide feedback and diagnoses.

How I built it

We used TecoGAN, a GAN trained to generate super resolution videos from low-res videos. Furthermore, we chose to use Flask to build our webapp, as it was quick and functional. We created multiple SQL tables to store user data and login information; we did so while keeping security and privacy in mind. Using our experience in web development, we packaged the application into a simple and intuitive interface.

Challenges I ran into

We did not have enough time to implement everything we envisioned, and therefore we had to prioritize and focus on core functionalities. It was not possible to train a full GAN with the short amount of time, so we had to use a model pre-trained on non-medical data. We were mostly limited by computation power. The TecoGAN utilizes lots of computational resources to infer on live video. So we had to settle for short video clips rather than transforming the entire video conference. We ran into difficulties with the chat functionality as well, but we persevered and completed a minimum viable product.

Accomplishments that I'm proud of

We are proud that we persisted through the many challenges and completed a final product. We were able to fully integrate the GAN model and generate higher resolution videos using our website, create convenient interfaces with chat and video viewing tools, configure user and video databases to store data safely, and design an intuitive UI to facilitate client-user interactions.

What I learned

Although there were loads of technical challenges on the way, the most important skill we learned was being able to collaborate virtually while keeping each other motivated through the night. We had an ambitious goal from the start and being able to work as a team to resolve the many technical/implementation issues that we ran into was the biggest learning opportunity for all of us.

What's next for DiagnoSEE

We want to adapt DiagnoSEE to be compatible with the current telemedicine platforms that are in use. Moreover, we hope to include more functionalities such as allowing physicians to annotate video frames for particular areas of interest; implementing a feedback loop between patient videos and the GAN model to improve model performance; adding video editing tools to adjust lighting, obstructions, etc. The current version is fully functioning with relatively light workloads. If DiagnoSEE were implemented in production, we plan on using distributed computing and dynamic scaling to increase our rendering capabilities. We also plan on implementing a flight control to manage the rendering queue, adding in parameters such as priority to give precedence to more timely appointments.