Inspiration
I wanted to find a healthcare dataset with a ton of missingness and ambiguity that would challenge me to not only apply what I learned as a Master of Data Science student but also build AI agents and use them to query the dataset.
What it does
My project analyzes a meningitis patient dataset to answer one of the most urgent questions in the condition’s early care: is this infection bacterial or viral? Bacterial meningitis is a medical emergency needing immediate treatment, while viral meningitis usually resolves on its own, and lab confirmation can take days. The project combines classification models that make predictions with AI agents that explain them.
Cleaned and validates messy medical data Assessed missing values with statistical tests (MCAR vs. MAR) and chose a handling strategy for each column, including dropping rather than imputing missing target values. Found and corrected impossible values, such as ages over 110 and a spinal fluid protein of zero. Uncovered hidden data leakage: three blood measurements almost perfectly encoded whether a pathogen was detected, so they were excluded from that model.
Tested what actually differs between patients Hypothesis tests confirmed that bacterial cases have significantly lower spinal fluid glucose and higher protein, white cell count, and CRP than viral cases, with large effect sizes. A logistic regression showed that the apparent link between diagnosis and death disappears after adjusting for other factors.
Developed classification models for pathogen detection as well as bacterial/viral diagnosis Pathogen detection. This model predicts whether a pathogen will be detected from spinal fluid glucose, protein, white cell count, CRP, age, and gender. I deliberately excluded three blood measurements after finding they almost perfectly encoded the answer, which would have produced a meaningless 100% accuracy. Of seven models compared, a random forest performed best (cross-validated ROC-AUC 0.96; held-out test ROC-AUC 0.94; about 86% accuracy). It outperformed logistic regression (ROC-AUC 0.89) because the relationships aren’t straight lines: risk changes sharply once glucose drops or protein rises past certain levels, and tree-based models capture those thresholds and interactions naturally. The forest also handles skewed lab values and outliers without needing scaling, and increasing its minimum leaf size reduced overfitting on the roughly 15% of patients whose labs fall in the overlap between groups.
Bacterial vs. viral diagnosis. This model answers the core clinical question: is a case bacterial or viral, based on spinal fluid labs alone? XGBoost achieved the highest cross-validated ROC-AUC (0.975), narrowly ahead of gradient boosting (0.973) and random forest (0.972). Boosted trees work well here because each new tree corrects the previous ones’ mistakes, refining the borderline cases where one lab looks bacterial and another viral. Hyperparameter tuning changed performance by less than 0.002, and every flexible model converged near the same score, showing the model had reached the limit of what the data supports. Because missing a bacterial case is far more dangerous than over-calling one, I lowered the decision threshold to prioritize recall for bacterial cases.
- Explains results with two AI agents built on Claude Risk Explanation Agent: answers questions like “Why was this patient rated High Risk?” in plain language, using SHAP to identify which lab values drove each prediction, and runs what-if scenarios. Unknown Diagnosis Agent: investigates the 136 patients with no recorded diagnosis. It found they are about three times more likely than diagnosed patients to have mixed or unusual lab patterns, and it builds a prioritized review queue of the least certain cases for clinicians.
How we built it
I built the project in Python in a Jupyter notebook, moving from data cleaning to statistical testing, then modeling, and finally the AI agents.
Data cleaning and validation Using pandas and NumPy, I profiled every column for missing and implausible values. For each column with gaps, I ran statistical tests (Mann-Whitney U and chi-square, via SciPy) comparing patients with and without missing values to determine whether the gaps were random (MCAR) or related to other variables (MAR). That guided how each was handled: median imputation for randomly missing labs, a model-informed approach for invalid ages, an “Unknown” category for missing gender, and dropping (never imputing) rows with a missing prediction target. I also flagged clinically impossible values, such as ages over 110 and a spinal fluid protein of zero, and documented each decision.
Exploratory analysis and hypothesis testing I visualized distributions and group differences with seaborn and matplotlib (histograms, boxplots, and correlation heatmaps), then tested formal hypotheses. Mann-Whitney U tests with Bonferroni correction compared labs between bacterial and viral cases, and between patients with and without a detected pathogen. Chi-square tests checked associations between categories, and a multivariable logistic regression in statsmodels tested which factors were linked to mortality after adjustment. I reported effect sizes alongside p-values throughout.
Classification models Using scikit-learn and XGBoost, I built a consistent pipeline for each prediction task:
I used a stratified train/test split, held out the test set, and used it only for final evaluation. Five-fold cross-validation on the training data to compare seven model types: logistic regression, decision tree, KNN, SVM, random forest, gradient boosting, and XGBoost. Hyperparameter tuning with grid and randomized search. I chose decision thresholds from cross-validated predictions to prioritize catching bacterial cases.
Along the way, a single-feature check revealed that three blood measurements almost perfectly predicted pathogen detection, so I excluded them as data leakage.
- AI agents I built two agents on the Anthropic Claude API using tool calling. Each has two layers:
An analysis layer in Python that does all the math: a trained model, SHAP values explaining each prediction, and functions that return patient explanations, what-if scenarios, exact counts, and review queues. An agent layer where Claude decides which tools to call, reads their results, and writes a plain-language answer. A system prompt sets guardrails: use only tool results, present reasons as hypotheses, report confidence, and never give a diagnosis or treatment advice.
I evaluated the agents by checking their answers against the data (every case ID, count, and lab value I spot-checked was correct) and fixed issues testing revealed, such as long answers being cut off and a missing tool for exact counts. API keys were kept out of the notebook using python-dotenv.
Tech stack: Python, Jupyter, pandas, NumPy, SciPy, statsmodels, scikit-learn, XGBoost, SHAP, seaborn, matplotlib, Anthropic Claude API, python-dotenv.
Challenges we ran into
The data cleaning process took a long time because I was very thorough in handling missing values in categorical and numerical columns. For the categorical columns outcome and pathogen_present, I dropped the null values, since I planned to use both as target variables in classification model pipelines. For the numeric columns Age, Protein_Level, Glucose_Level, Hemoglobin, WBC_Count, WBC_Blood_Count, Platelets, and CRP_Level, I filled null values with each column's median. I also ran into output issues with the agents I ran, where the answers were too long to see in my notebook. I had to debug the issue and output new, better-written agent files that produced the answer I wanted.
Accomplishments that we're proud of
I built my first two agents for the data science lifecycle to help me better understand the data. I am so happy! I am also really happy with the performance of the classification models that I built
What we learned
About the data
Spinal fluid labs strongly separate bacterial from viral meningitis. Bacterial cases had markedly lower glucose (median 25 vs. 60 mg/dL) and higher protein, white cell count, and CRP, all with large effect sizes. Models using these labs alone reached a ROC-AUC of about 0.97, matching the patterns clinicians rely on before lab results are confirmed. Cases with no recorded diagnosis aren’t just unlabeled cases. The 136 “Unknown” patients were about three times more likely than diagnosed patients to have mixed or unusual lab patterns, most often bacterial-looking inflammation with normal or high glucose. That ambiguity is a plausible reason no diagnosis was recorded. Survival was essentially unpredictable from this data. No lab value, age, or test result was significantly associated with death, and the apparent link between diagnosis and mortality disappeared after adjusting for other factors. Knowing what a dataset can’t answer is a result in itself. Risk level reflects severity, not mortality. High Risk patients were mostly bacterial cases, but their death rate was similar to Low Risk patients’.
About building trustworthy models
A perfect score is a warning sign. Our first pathogen model was nearly 100% accurate because three blood measurements encoded the answer. Catching that leakage mattered more than any tuning. Every model hit the same ceiling. Seven very different models converged on similar performance, and tuning changed results by less than 0.002. That told us the limit was in the data, not the algorithm, so effort was better spent on features, thresholds, and interpretation. Cleaning decisions should be tested, not assumed. Testing why values were missing (random or not) changed how we handled them, and showed that some invalid ages needed a different approach than the simple median. Choose metrics and thresholds based on which mistake costs more. Missing a bacterial case is far worse than over-calling one, so we tuned thresholds for bacterial recall rather than overall accuracy.
About building AI agents
Agents are only as reliable as their tools. Keeping all calculations in tested Python functions, with Claude explaining the results, kept the agents accurate. Every case ID, count, and lab value we checked was correct. Guardrails make agents useful, not just safe. Requiring the agents to report confidence, present reasons as hypotheses, and recommend clinician review made their answers more trustworthy to a non-technical reader. Evaluate agents the way you’d evaluate a model. Testing revealed issues we hadn’t anticipated, such as long answers being silently cut off and questions the tools couldn’t answer exactly, and each became a concrete fix.
A note on the dataset: several patterns, including impossible hemoglobin values, blood counts generated from the target, and no link between age and mortality, suggest this is a synthetic teaching dataset. Our findings demonstrate the approach, but would need validation on real clinical data before informing care.
What's next for Analyzing Meningitis Diagnoses with AI and Machine Learning
I would like to spend more time building another agent focused on deceased patients to determine which factors might have been the most correlated with their passing I would like to build a regression pipeline using pathogen features such as Glucose_Level, CRP_Level, WBC_Count, Protein_Level, as well as categorical features including Gender_Female and Gender_Unknown to attempt to predict the patient age as the target variable.
Log in or sign up for Devpost to join the conversation.