Inspiration

We have known each other since fourth grade, long before we imagined building a medical computer-vision platform together. Rishin hopes to pursue gastrointestinal surgery, while both Kiran and Rishin are interested in how software, artificial intelligence, and clinical medicine can intersect to improve patient care.

ScopePilot was inspired by a deceptively difficult problem: colonoscopies generate thousands of video frames, yet physicians must simultaneously navigate the colon, identify subtle lesions, operate instruments, document findings, and prepare an accurate report. Small or visually ambiguous polyps can be difficult to distinguish from folds, debris, glare, or poor bowel preparation.

We wanted to build an intelligent “second set of eyes” that supports the physician without attempting to replace their clinical judgment.

What ScopePilot Does

ScopePilot transforms recorded or de-identified colonoscopy video into a structured, physician-reviewable procedure timeline. Its computer-vision pipeline detects and tracks suspected polyps and instruments, estimates anatomical location, captures representative images, and organizes these findings into a report.

Every AI-generated finding can be confirmed or dismissed by the physician. Confirmed findings can then appear in a procedure report containing timestamps, anatomical locations, biopsy records, and curated screenshots. Physicians can also create procedure-scoped patient credentials, allowing a patient to view only their finalized timeline and report through a dedicated portal.

ScopePilot is a research and hackathon prototype—not a clinically validated medical device—and is designed for synthetic, public, or de-identified demonstration data.

How We Built It

We built ScopePilot as a full-stack clinical workflow rather than an isolated machine-learning notebook.

The frontend uses Next.js, React, TypeScript, Tailwind CSS, and Radix UI. Its navigation and dashboard are organized around the steps of a colonoscopy: creating a procedure, selecting its indication, uploading video, reviewing detections, logging biopsies, confirming findings, and publishing a report.

The FastAPI backend provides session authentication, role-based access control, procedure management, video processing, WebSocket updates, audit logging, report generation, and procedure-scoped patient access. SQLite and Alembic manage structured clinical data, while videos, screenshots, model weights, and exports are stored separately on the filesystem.

Our vision pipeline combines several specialized models:

• A YOLO model for polyp and lesion detection.
• A YOLO model for instrument detection.
• A ConvNeXt-based classifier for anatomical localization.
• Temporal tracking and confidence smoothing to reduce duplicate and unstable detections.

Instead of treating every frame as an independent event, ScopePilot associates detections over time. Bounding-box similarity is measured using intersection over union:

[ \operatorname{IoU}(A,B)=\frac{|A\cap B|}{|A\cup B|} ]

When detections remain spatially and temporally consistent, the system treats them as one candidate finding rather than repeatedly logging the same polyp. This temporal reasoning is essential because a single lesion may remain visible across dozens of frames.

We trained and evaluated models using public gastrointestinal datasets, including Kvasir-SEG for lesions, Kvasir-Instrument for endoscopic instruments, and CAS-Colon-derived data for anatomical classification.

Challenges We Faced

The greatest challenge was learning that medical computer vision is not simply an image-classification problem. Reflections, motion blur, fluid, debris, variable lighting, bowel folds, and poor preparation can all resemble pathology. Early versions produced unstable detections and could populate the dashboard with findings that were not supported by the video.

We mitigated this with confidence thresholds, consecutive-frame requirements, temporal deduplication, anatomical smoothing, and physician confirmation controls. These safeguards do not eliminate model error, but they prevent uncertain frame-level predictions from automatically becoming clinical records.

Anatomical localization was particularly difficult because neighboring colon segments can look extremely similar. Our model improved during training, but confusion between visually related regions—especially the flexures—showed us why temporal context and physician override controls are necessary.

Deployment introduced another layer of challenges. Computer-vision applications require large dependencies, model weights, persistent storage, and native Linux libraries that ordinary web applications do not. Moving ScopePilot from a local Mac environment to Railway required resolving Python dependencies, configuring OpenCV runtime libraries, and designing persistent storage for weights, videos, screenshots, reports, and the database.

What We Learned

We learned that trustworthy clinical AI depends as much on workflow engineering as model accuracy. A prediction is only useful when it is traceable, reviewable, and presented at the correct point in a physician’s workflow.

We also learned the importance of dataset balance, representative validation data, confidence calibration, and temporal reasoning. Improving a model is not only about reducing training loss; it requires understanding which classes fail, why they fail, and how those errors affect users.

Most importantly, ScopePilot taught us to treat artificial intelligence as decision support. The physician remains the final authority, while the software reduces documentation burden, organizes procedural evidence, and draws attention to findings that deserve review.

What’s Next

We want to expand ScopePilot beyond polyps to findings such as diverticulosis, hemorrhoids, inflammation, fistulas, and other gastrointestinal abnormalities. We also hope to connect endoscopic appearance with pathology and histology data so future models can provide carefully calibrated risk estimates based on morphology, size, color, and surface pattern.

Longer term, we envision integration with clinical reporting systems, stronger anatomical sequence modeling, larger multicenter datasets, and prospective validation with medical professionals. Our goal is to develop ScopePilot into a safer, more explainable, and clinically useful assistant for both physicians and patients.

Built With

• alembic
• fastapi
• google-colab
• next.js
• node.js
• numpy
• opencv
• pillow
• pydantic
• pytest
• python
• pytorch
• radix-ui
• react
• shadcn/ui
• sqlalchemy
• sqlite
• tailwind-css
• timm
• torchvision
• typescript
• ultralytics-yolo
• uvicorn
• vitest