GoldenbAIR: Pulmonary Auscultation Assistant

Inspiration

Our project was inspired by a need in the healthcare industry for clinicians, EMT specialists, and at home care providers to identify pulmonary disease through lung patient data from stethoscope audio files converted into spectrogram graphs.
At the moment, identification is difficult due to inconsistency in artifacts for identification, a need for a highly trained ear, and human error in identification
As Bioengineering and EECS major students, we are committed to identifying a way to combat the effects of pulmonary distress which is currently a leading killer in pediatric patients.

What it does

*We designed a website which takes in patient lung audio files, converts them into spectrogram graphs using Fourier transforms, identifies peaks to have significant identification features to train our model, uses an MLM with ResNet 18 to specifically weight different parameters, and outputs a most likely underlying pulmonary diagnoses based on spectrogram graph data. We then incorporated a Gemini API AI assistant which took in the diagnoses outputted from the MLM and provided follow-up action items for the clinician to diagnose with complete confidence based on severity of condition and possible next steps.

How we built it

(1) Feature engineering: Using publicly available patient data of breathing audio, we created spectrograms, a frequency and time plot of the audio sounds, via FFT and convolutional analysis. We then constructed an algorithm to identify peaks in the spectrograms which are artifacts in breathing. From this, we extracted features to train our model on, including number of transients/peaks, breathing and transient intervals, and presence of wheezing, crepitus, and crackling.

(2) Model and Training: We built and trained a ResNet18 model on the features extract from signal processing, which is an 18-layer CNN with back-propogation and Adam optimizer for parameter weight computation.

(3) Patient Clinician Interface: We created a dashboard where patients and clinicians can upload audio files using Streamlit. This is integrated with our model and Gemini, where a Gemini API is called to generate and AI care summary to advise patients on next steps for care.

Challenges we ran into

Throughout the process of building and testing our MLM algorithm and designing our website, we faced numerous challenges which we worked to overcome in order to successfully build our product.

Accurately and consistently identifying peaks from lung audio file data which we had Fourier transformed into spectrogram data
Identifying artifacts such as wheezing, crackles, and crepitus throughout our spectrograms to differentiate them from normal breathing sounds and general background noise
Identifying an algorithm to accurately predict disease diagnoses using pattens in spectrogram data and peak identification (increased accuracy could be found with more diseased patient data)

Accomplishments that we're proud of

From the start, we knew that finding apt medical data for us to be able to train a model on would be difficult. We were able to use some of our clinical judgement to assess the validity of this data and to identify different features that we could use in our model. In addition, another major accomplishment of ours was being able to identify the peaks of our spectrograms and to isolate features that would be unique to each diagnosis and important and to train a model with a >70% diagnosis accuracy. Finally, we were able to integrate Gemini 2.5 Flash to further the usability of our product in generating explanations of the disease and potential treatment plans and next steps.

What we learned

This project allowed us to really put together so many concepts across various classes and learn how to use some amazing development tools along the way. From learning now to use git hub without instructor support to fully integrating both back end and front end development this project taught us how to be independent developers. On the more conceptual side, this project taught us to be independent developers. On the more conceptual side this project allowed our team to delve into the process of building and training an ML model with various data modalities from images to audio recordings.

What's next for GoldenbAIR: Pulmonary Auscultation Assistant

Refining the MLM with more patient data to increase the number of disease of website can assist with would allow for a more robust care plan for patients. We would also like to further develop the front end after consultation with clinicians and health care providers as to what information would be most helpful to them. Further features we wish to implement could also be incorporated and we can adjust the API accordingly. Additionally we would like to implement a patient interface in which recordings may be taken and submitted for Telehealth care.

Please visit our table!