Lumen Colonoscopy AI

Website Main Page
Polyp recognition and likelihood estimate mapped to image (yellow - intermediate probability of being polyp)
Polyp recognition and likelihood estimate mapped to image (red - greater likelihood of being polyp)
Training loss curve for Polyp identification
Confusion Matrix for Malignant Polyp detection Model
Malignancy detection model for benign vs malignant
Malignancy detection model loss curves for benign vs malignant
Use of Open Swarm to spawn and run multiple agents
Precision - Recall curve for one vs rest for all images

Team Name: Lumen Colonoscopy AI

Inspiration

A gastroenterologist doing a morning list of colonoscopies has to answer two questions about every polyp within seconds: how big is it, and could it be cancer? After every case comes a report. Doctors spend nearly two hours on paperwork and computer records for every hour they spend with patients, and about one in four polyps is missed during colonoscopy. Delay in diagnoses due to testing times can also worsen colon cancer in vulnerable patients.

The information that goes into a polyp report is scattered. The image is on the screen, the measurements are in the doctor's head, and the guidelines and research papers are somewhere else. Track 1 of the HealthLink Hackathon asked how Open Swarm could help with colonoscopy. Our idea was to let trained AI models produce the objective findings and let a team of Open Swarm agents turn them into a referenced draft report, so the doctor reviews and signs instead of writing from scratch.

What it does

Lumen is a web app. The doctor uploads a still image from a colonoscopy and picks one of two analyses.

Polyp Detection and Sizing: the AI outlines the polyp and estimates its size, giving a likely range.
Malignancy Risk Assessment: the AI says whether the lesion looks benign or malignant, and shows a heatmap of the part of the image it focused on.

Lumen then hands the case to Open Swarm. The agents read the AI results, look at the images, search PubMed for relevant research and write a structured clinical report. The report covers the findings, the polyp's shape classification, a referenced assessment, points for the doctor to check, suggested next steps and a sign-off section. Lumen turns that report into a PDF with the image included. Nothing is final until the doctor signs it.

How we built it

For polyp outlining, we trained a U-Net, a type of image model built for drawing outlines. We started from a model pretrained on everyday photos and trained it on the Kvasir-SEG dataset: 800 images for training and 200 kept aside for testing.

Measuring size from one image is hard because there is no ruler in the picture. We modelled the colonoscope's wide-angle lens and assumed the camera is between 5 and 20 millimetres from the polyp. That gives a size range instead of one number. For example, Lumen might report about 14 mm, likely between 7 and 28 mm, and show every size category the range covers.

For malignancy, we trained an EfficientNet, a compact image classification model, on two public datasets, GastroVision and ERCPMP. Cancer images were much rarer than benign ones, about one in eight, so we gave the cancer images more weight during training so the model would not simply ignore them.

For the report, our web app (built with FastAPI) saves the AI results for each case together with a task file for the agents. The task file points Open Swarm to a report template and a set of rules: write in plain clinical language, describe likelihood in words instead of numbers, never mention model names or scores, and cite only real research. We also built a tool server that gives the agents two tools: one runs the polyp outlining model, and one searches PubMed and returns article summaries. Once the agents finish the report, the web app picks it up automatically and offers it as a PDF download.

Summary of agents used: 1 agent is used for assessing the polyp image 3 agents are used for literature focusing on morphology (Paris classification, sessile lesions), polyps 10 mm and larger (advanced adenoma risk, surveillance guidelines) and resection (ESGE polypectomy guideline, cold vs. hot snare) 2 agents are focusing reviewing. 1 focuses on verifying and another focuses on clinical safety and avoid overclaiming.

We trained everything on a single laptop graphics card, using PyTorch.

Challenges we ran into

A model we had to drop: we first tried to predict the exact polyp type from the image, such as adenoma or hyperplastic polyp. It was not accurate enough on standard colonoscopy images, so we removed it. Shipping something unreliable would have been worse than shipping nothing.
Size without a ruler: any size in millimetres from a single image is an estimate. We chose to show that uncertainty openly instead of hiding it.
Keeping the agents trustworthy: without guardrails, AI writers can invent references or copy raw model numbers into the report. Strict rules, a fixed template and a real PubMed search tool kept the reports grounded and readable for doctors.
Hand-off between the app and the agents: the app had to tell the difference between a report still being written and a finished one. It waits until the report file has stopped changing for a few seconds.
Setup issues: our new graphics card needed a newer version of PyTorch, and we worked around some Windows-specific problems when loading the data.

Accomplishments that we're proud of

The polyp outlines match expert drawings very closely, with an overlap score of 0.91 out of 1 on 200 test images the model had never seen.
The malignancy model scored 0.91 out of 1 on a standard measure of how well it separates benign from cancerous lesions. It caught about 4 in 5 cancers and correctly cleared about 9 in 10 benign polyps.
We evaluated honestly: the test images were never used to choose or tune the models, and an extra set of images is used for validation.
Lumen works end to end: upload an image, get AI results in seconds, and let Open Swarm agents produce a referenced PDF report for the doctor to sign.

What we learned

Being open about uncertainty builds trust. Doctors prefer a size range with clear assumptions over a single number that only looks precise.
Agents work best with structure. Open Swarm was most useful when we gave it a focused task, clear rules, a template and the right tools. Then the agents can do the research and drafting that would otherwise take up a doctor's time.
Knowing when to drop a feature is part of building something trustworthy.
We also got to learn a lot from speakers in the startup ecosystem, particularly about clinical trials and got to learn about OpenSwarm.

What's next for Lumen Colonoscopy AI

Testing with gastroenterologists on real cases.
Support for narrow-band imaging, a special light mode used during colonoscopy.
Live video analysis and detection of several polyps in one image.
Connecting Lumen to hospital reporting and patient record systems. -Scaling to multiple image analysis use cases, such as histopathology or radiology.

Lumen is a hackathon research prototype, not a medical device. It assists doctors and does not provide a diagnosis.