SmearSight

An OpenCV 5 agent that reads malaria blood films from a phone on the lab's own microscope, stops when the treatment band is decided, and asks the scientist only about the cells that matter.

Comment

Live: https://16-146-1-219.sslip.io · Code: https://github.com/Osiyomeoh/smearsight · Technical report: https://github.com/Osiyomeoh/smearsight/blob/main/docs/REPORT.md · Video (4:59): https://youtu.be/xTK9bDzRk5M

Inspiration

Malaria is curable, yet in 2024 it killed about 610,000 people, most of them young children in Africa: about 1,200 children under five every day (WHO, World Malaria Report 2025). Where treatment is decided, it is still diagnosed by a person counting red cells under a microscope, field after field, and a busy lab often has to skip fields. Whether a patient's treatment changes depends on what share of those cells are infected.

What it does

SmearSight sees, reasons and acts:

  • See. Every field passes an OpenCV 5 quality gate (focus, exposure, noise, stain). OpenCV evens out the lamp, normalises the stain and separates touching cells with watershed, and a classifier in OpenCV's DNN module calls each cell. A conformal set flags the cells it isn't sure about.
  • Reason. The agent turns the counts into a parasitaemia band (below 2%, 2–5%, above 5%) with a 95% interval. It reads sharp, cell-dense fields first and stops as soon as the band is decided.
  • Act. On a borderline slide it asks the scientist only about the cells whose call could change the band. If the band is still undecided, it refers the slide to a second reviewer and says why. A person signs every slide.
  • Live mode coaches the technician holding the phone at the eyepiece: focus, fix the lamp, move on. It never counts the same field twice and says when to stop.
  • MCP. Any MCP client can check a field, read a slide or fetch the lab's rates. The stopping and escalation rules are inside the tools, so a language model can't talk it into stopping early.

Results

  • A hospital it never saw: 187 held-out patients at Chittagong Medical College Hospital, photographed by phone through the eyepiece, every red cell marked by an expert (NIH-NLM). It was set up from six patients' taps, the way a lab would do it. Where its interval sits inside one band, it agrees with the expert on 95.5% of patients, and all 31 patients it placed above 5% really are above 5%.
  • Agent stopping: across virtual slides it matches the accuracy of reading every field with 16% fewer fields.
  • Ablation: each step was switched off in turn. Without stain normalisation, band accuracy falls by 40 points.
  • A warm lamp: 929 of 935 hospital fields pass the quality gate, and a really faded stain is still caught on 98% of fields.
  • Cost: 257 ms a field and $1.43 per 1,000 slides on AWS Graviton4, 22% less than x86.

How we built it

Python 3.12 and OpenCV 5.0 (pinned), with the classifier as ONNX in cv2.dnn. A FastAPI service with an MCP endpoint runs on AWS Graviton4 (c8g). /api/version reports the commit, the OpenCV build, the CPU features and the hash of every model. CI runs on x86 and Arm, there is a Dockerfile, and server/deploy.sh deploys it.

COOL: we ran the whole pipeline on COOL v3 (Graviton-optimised OpenCV from the AWS Marketplace) against stock OpenCV 5.0 on the same box. COOL ran CLAHE in 37% less time, contours 19% and blur 16%, but was 8% slower end to end, so the server ships the faster build, stock OpenCV 5.0. OpenCV 5's DNN runs our classifier 1.5× faster than OpenCV 4.13 on Graviton4. All raw results are in results/cool/.

Challenges we ran into

  • Warm lamps turn the glass orange, so a colour check on raw pixels saw no stain. We now read the stain against each field's own glass.
  • Moving to a new country, lab, microscope and phone. Per-site calibration and spot checks refit the detector's rates for each lab.
  • Knowing when to stop: the stop is tied to the clinical band, not to a fixed cell count.

Accomplishments we're proud of

A test on real phone photos from a hospital it had never seen, and an agent that refers the hard slides instead of guessing. It runs for about a seventh of a cent a slide.

What we learned

Most of the clinical value is in deciding what not to read, and in what to ask a person.

What's next

Live mode at a real microscope in a partner lab, more sites calibrated from spot checks, and thick-film support.

Decision support for laboratory scientists, not a diagnostic device. Limitations and failure cases: REPORT §6.

Credits

Blood-film photos and expert marks: courtesy of the U.S. National Library of Medicine (Kassim et al., IEEE JBHI 2020). Photos of people: U.S. President's Malaria Initiative (public domain), used without implied endorsement.

Built With

Share this project:

Updates

Submission history