Clinical Model Tuner

Screenshot of our web application

Inspiration

We are passionate about improving the efficiency and effectiveness of the healthcare system. With the recent FDA approval of decision-making neural networks for use in clinical trials, one of the major requirements is the adaptability to the diverse domains of data across the institutions. A neural network must be device-agnostic to be considered for such trials. To further explore methods of increasing clinical confidence in artificial intelligence decision making in the clinical space, we have developed an intuitive tool that allows clinicians to fine-tune networks to the domain of the dataset. We've attached our presentation to the google drive link found below.

What it does

Clinical Model Tuner allows the user to easily fine-tune a model to a batch of labeled sample data. Currently, it is fully functional for classification problems and can be adapted to other tasks. Clinical Model Tuner is a web application (check out our functional proof of concept in the link attached below). where the clinician can provide a model to be tuned through a Google drive link. The clinician then selects the images and classes to fine-tune the model with and provides an initialization script that details the preprocessing steps (easily replicated by the creator of the model. Important parameters such as the number of epochs and the number of layers to freeze are also adjustable by the user. Our tool repeatedly improve classification performance by 15-30% across two different classification problems. Test it out with the test data stored in google drive (accessible through the link below).

How we built it

We prepared two datasets to test the framework on - Pneumonia classification: link and retinal disease detection: link. Gaussian noise and intensity/contrast adjustments were introduced to the data to imitate the extreme spectrum of noise that one might expect in a poor acquisition environment. We utilized PyTorch to perform fine-tuning and model evaluation based on the parameters specified. The backend was developed using Node.js + express to create a REST API server for communication with the react.js frontend. Furthermore, implementing an RPC connection, allows us to create a complete decoupled backend system with a dedicated machine learning server.

Challenges we ran into

As is the case with many data-focused projects, we had trouble finding a suitable dataset that had pre-trained networks. We had the option of training networks on the data from scratch, but feel that this best represents the real-world application of this web application. The scope of the project was also quite large, so training two individual models that performed up to par with Kaggle submissions to have a fair comparison was not within our time budget. PyTorch was also a relatively new library for us as our work was mainly on TensorFlow and Keras.

Accomplishments that we're proud of

We are proud to create an easy-to-use web application in two days. The fact that Clinical Model Tuner is able to fine-tune pre-trained models to comparable performances is an accomplishment that we are proud of. Yet, we are the proudest of the potential that this project shows - perhaps fine-tuning of a model to the domain of the client institution or the domain adaptation of client data to the original trained domain could be techniques that would increase the confidence of clinicians in decision-making AI.

What we learned

Besides programming with new libraries, expectation and time-management are essential to hackathons. The time-crunching stress of hackathons really stresses the importance of these skills that we know will carry through to other aspects of our lives.

What's next for Clinical Model Tuner

If time permits, we would be interested in exploring this project further. The performance of this two-day project shows the potential of different techniques we could explore to increase confidence in automated clinical decision-making. We would also be interested in exploring the adaptation of the clinical real-use data domain to the domain of training data.