Inspiration

I was diagnosed with Crohn's disease in 7th grade, after more than a year of not being able to get a diagnosis. Looking back, it makes sense that it took so long. Crohn's symptoms overlap with a lot of other conditions, there's no single test for it, and it often shows up in the small intestine, which a regular colonoscopy can't reach.

Capsule endoscopy is one way doctors look there. You swallow a camera the size of a pill, and it takes tens of thousands of pictures on its way through. Then a doctor has to go through all of that footage, usually 30 to 60 minutes per study, and small lesions are easy to miss. I can't fix everything that makes diagnosis slow. This was one piece I could actually work on: getting doctors to the frames that matter faster.

What it does

You upload capsule frames or a full video. The app flags frames that likely show Crohn's lesions and sorts them so the most serious ones (stenosis and large ulcers) come first. Each flagged frame gets a Grad-CAM heatmap showing which part of the image the model based its decision on.

There's also a timeline of the whole study with every frame colored by severity, so you can see where the disease clusters. A sensitivity slider lets you trade fewer frames to review against more lesions caught, and it shows the expected numbers for each setting from held-out testing. For video, each frame's score is smoothed over 3 seconds, because a real lesion stays in view for a while. You can export everything as a PDF report. It's a research prototype. A gastroenterologist still makes the diagnosis.

How we built it

I trained on CrohnIPI, a public dataset of 3,484 capsule frames from 66 patients, labeled by gastroenterologists at CHU de Nantes as normal or one of six lesion types. The model is EfficientNet-B0, pretrained on ImageNet and fine-tuned in PyTorch on Google Colab.

The data is lopsided: 61% of frames are normal and only 3.6% show erythema. I weighted the loss by class frequency so the model wouldn't just call everything normal. I also skipped color augmentation on purpose, since redness is the main clue for erythema. Training at the images' full 320 × 320 resolution instead of 224 × 224 raised the lesion-vs-normal AUC from 0.946 to 0.962.

To see how it holds up on equipment it never saw, I ran the final model on Kvasir-Capsule, footage from a hospital in Norway recorded with an Olympus capsule instead of a PillCam. No retraining.

For video, each frame's lesion score is averaged with its neighbors:

$$\bar{s}_t = \frac{1}{w}\sum_{k=t-\lfloor w/2 \rfloor}^{t+\lfloor w/2 \rfloor} s_k$$

where w covers 3 seconds. On a sample clip with expert-marked ulcers, this raised the share of ulcer moments that got flagged from 55% to 78%.

The interface is Gradio. Grad-CAM comes from pytorch-grad-cam, video decoding from OpenCV, and the PDF from ReportLab. You can run the whole thing from the GitHub repo in Colab with one click.

Challenges we ran into

My first results were too good, and it took me a while to figure out why. A capsule camera takes several photos per second, so one ulcer shows up in a run of nearly identical frames. The dataset's suggested splits scattered those near-copies across training and testing, so the model was basically being quizzed on pictures it had already studied. Once I kept each run of consecutive frames on one side of the split, balanced accuracy fell from 0.737 to 0.627. That lower number is the real one.

Erythema and edema were the hardest classes. They're rare and subtle, and edema kept getting mistaken for nearby ulcers. For a triage tool, though, calling an edema frame an ulcer is much less of a problem than missing it entirely, and the model flagged about 9 in 10 lesion frames as some kind of lesion.

The Olympus test humbled me a bit. AUC dropped to 0.852. Ulcers held up (90% caught), but erythema fell to 56%, probably because the Olympus camera makes tissue look more orange than the PillCam does. The slider setting that caught 95% of lesions on PillCam caught only 78% on Olympus.

Accomplishments that we're proud of

Catching the data leakage is the thing I'm proudest of. It would have been easy to report 0.737 and move on. I'm also glad I tested on a second dataset from a different country and camera, since a lot of projects (and published models) only report results on the data they trained on.

What we learned

How you split the data can matter as much as the model. With frames pulled from video, a random split quietly inflates your results.

A model that works at one hospital won't necessarily work the same at another. A different camera was enough to shift the numbers, so a tool like this would need to be checked and recalibrated before being used somewhere new.

I also stopped thinking about plain accuracy. For triage, what matters is how many lesions get caught and how much footage the doctor still has to look through, so I built the app around that trade-off.

What's next for Crohn's Capsule Triage

  • Calibrating the sensitivity settings for each capsule brand
  • Training on data from more hospitals, which is the biggest limit right now
  • Grouping neighboring flagged frames into single findings, so a doctor sees "6 findings" instead of "40 frames"
  • Getting a gastroenterologist to review the heatmaps and test the app on full-length studies

Built With

Share this project:

Updates

Submission history