SuaraNafas

Multimodal AI screening for Tuberculosis.

Comment

Inspiration

Tuberculosis (TB) remains the leading infectious disease killer worldwide. Yet, what is even more alarming is the "missing millions"—patients who are never diagnosed and unknowingly continue to spread the disease.

Currently, the initial screening process heavily relies on self-reported cough questionnaires. According to the WHO (2021), this subjective method is severely unreliable, yielding a sensitivity of only 42%. This fails to meet the WHO targets for triage testing. While digital chest X-rays offer a more accurate alternative, they are completely out of reach for resource-limited primary healthcare facilities.

This inspired us to rethink the screening process through the lens of Acoustic Epidemiology. We wanted to objectify triage testing by treating a cough not just as a symptom, but as data. TB-related coughs have specific acoustic signatures—distinct frequency patterns, intensity, and rhythms—that differentiate them from other respiratory conditions. Our vision was to turn this acoustic signal into a fast, non-invasive diagnostic tool that runs on standard smartphones.

What it does

SuaraNafas is an AI-powered TB screening web application designed for community-based triage testing. While it is not a replacement for a doctor or a final diagnosis, it serves as a highly accessible first screen.

Users are greeted with a clean, premium interface. To test their TB risk, they simply tap "Start Recording" and cough into their phone's microphone. The app captures the raw audio, ensures clear acoustic properties, and securely submits it to our backend for analysis.

Within seconds, the user receives a TB risk assessment (TB-positive, TB-negative, or Indeterminate). If flagged as high risk, the app seamlessly integrates with SatuSehat (the Indonesian Electronic Medical Record system) to help the user schedule a clinic visit immediately. To ensure our AI isn't a black box, we integrated an Explainable AI (XAI) dashboard powered by an LLM chatbot. This assistant breaks down the complex prototype similarities and clinical feature contributions into plain, reassuring language for the user.

How we built it

We developed the predictive model using a multi-country dataset from the "CODA TB Dream Challenge," which includes solicited cough samples and clinical metadata from 1,105 participants.

  • Audio Processing: We use librosa to extract Mel-spectrograms and 16 handcrafted acoustic features from raw audio.
  • Machine Learning Architecture: Our Multimodal SuaraNafas model is built in PyTorch. It processes the spectrograms using a ResNet18 backbone as a feature extractor. We utilize Attention-based Multi-Instance Learning (MIL) to handle multiple cough recordings per patient. For explainability and localization, we integrated ProtoPNet for prototype-based acoustic pattern matching and YOHO (You Only Hear Once) to detect the precise temporal boundaries of cough events. The final risk probability is derived by fusing these acoustic representations with normalized clinical metadata.
  • Backend: We serve the model inference and structured XAI endpoints via a FastAPI server, deployed on Hugging Face Spaces.
  • Frontend: The user interface is built with Next.js (App Router), TypeScript, and Tailwind CSS.

Mathematical formulation for our prototype matching (ProtoPNet): Given a latent patch \(z_{i,j}\) and a learned prototype $p_k$, the distance is computed as: $$ d(z_{i,j}, p_k) = \lVert z_{i,j} - p_k \rVert_2^2 $$ The similarity score $S$ is then calculated using a Gaussian RBF kernel to ensure explainable acoustic matches: $$ S = \max_{i,j} \exp \left( - d(z_{i,j}, p_k) \right) $$

Challenges we ran into

  • Data Quality & Imbalance: The dataset contained a mix of high-quality and noisy recordings. Some participants had missing audio files or corrupt WAV files. We had to implement rigorous preprocessing pipelines that dynamically swept through subfolders, handled missing mappings, and deliberately halted training rather than silently replacing corrupted audio with \(np.zeros\), which previously caused our BatchNorm variance to collapse (\(\sigma^2 \to 0\)).
  • Edge Portability vs. Complexity: Balancing the complexity of a multitask CNN (ProtoPNet + YOHO) with the need for fast, lightweight inference was difficult. We heavily profiled the MACs and CPU latency to ensure the architecture could hypothetically run efficiently on resource-constrained devices in the future.
  • Explainability: Medical AI requires trust. Translating raw model outputs (like YOHO boundary boxes and ProtoPNet similarity arrays) into a user-friendly format was challenging until we bridged the gap using an LLM to interpret the structured XAI JSON.

What we learned

  • We gained a profound understanding of Acoustic Epidemiology and how to fuse disparate modalities (tabular clinical metadata + raw audio arrays) into a unified PyTorch architecture.
  • We learned the hard way about the dangers of silent failures in deep learning data loaders. If corrupt audio is fed as zeroes, it destroys the variance in downstream batch normalization layers, causing catastrophic performance collapse.
  • We discovered how powerful Explainable AI (XAI) can be for user experience. Providing a "black box" percentage is scary; providing a conversational AI that says, "Your cough matched specific frequency patterns, but your risk is mostly elevated due to your reported fever," builds trust and encourages users to seek actual medical help.

Built With

Share this project:

Updates