Inspiration
Medical students use surgical video to train, but raw surgical video itself is hard to scrub through. An operation can run for hours, and the moment a student actually needs might last seconds. Hands-on practice is even harder to come by. Surg+F makes surgical learning more accessible by letting students find the exact moment they need, then practice it themselves.
What it does
Surg+F is Ctrl+F for surgical video, built for medical students. It labels every second of an operation with the instruments, actions and anatomy on screen. Students can ask questions in plain words, jump straight to the right moment, study each step with a timeline and quiz, and practice the procedure on an interactive 3D model.
How we built it
Machine learning: We ran SurgVLP, a vision model pretrained on surgical video, over 50 real operations from the CholecT50 dataset to turn every second into features. On top of those features, we trained a neural network that predicts 29 labels at once (instruments, actions and anatomy) and evaluated it on surgeries it never saw. A Faster R-CNN draws experimental bounding boxes over the video.
Databricks: Extraction and training ran as serverless jobs, with every run tracked in MLflow and all data stored in Unity Catalog Volumes. A Lakeflow job loads nearly 13,000 labeled and predicted clips into a Delta table. Search runs on Databricks SQL, a custom Genie space answers free-form questions, and Llama 3.3 on Model Serving routes chat messages to the right tool. The whole app is hosted on Databricks Apps.
Frontend and backend: A plain JavaScript frontend with Three.js for the 3D practice model, and a Python FastAPI backend.
Challenges we ran into
- GPUs weren't available, so we re-engineered feature extraction to run on serverless CPU across all 50 surgeries.
- The dataset shipped as still frames, so we rebuilt and time-aligned the full-quality videos ourselves.
- Rare instruments and anatomy appear for only seconds, which makes them hard to learn; tripling our training data helped a lot.
- We ran out of free-tier Databricks compute the night before the demo.
What's next for Surg+F
We want to expand beyond a single procedure and improve detection of rare moments. The same second-by-second index could also auto-draft operative notes and serve billing, quality review and patients.
