Inspiration
During surgery, anaesthetists continuously monitor vital signs such as heart rate, SpO₂, blood pressure, EtCO₂, temperature, and respiratory rate. Yet documenting these values can still require repetitive manual transcription from the monitor.
We wanted to solve this without replacing or modifying existing medical equipment. VITAL was inspired by a simple idea: if a camera can see the monitor, software should be able to understand what it displays.
What it does
VITAL is a camera-based anaesthesia vitals digitization system that converts information displayed on an existing anaesthesia monitor into a structured digital record.
It captures the monitor screen, detects and extracts vital-sign regions, uses OCR and an optional ONNX-based CNN to recognize the values, and streams HR, SpO₂, NIBP, EtCO₂, temperature, and respiratory rate into a live dashboard.
VITAL also validates extracted readings and flags uncertain or physiologically implausible values instead of silently accepting them. Clinicians can review and correct flagged readings before signing the final record.
How we built it
The vision pipeline was built using Python, OpenCV, Tesseract OCR, and an ONNX CNN, with FastAPI providing the backend and SQLite handling local persistence.
The frontend was built with React, TypeScript, Vite, Zustand, Tailwind CSS, Framer Motion, and uPlot.
The system includes monitor calibration, live vital visualization, validation and alerting, review and sign-off, audit history, and PDF record generation. The architecture is designed to run fully offline, keeping captured data on the local device.
Challenges we ran into
The biggest challenge was making OCR reliable on a monitor rather than a clean document. Camera images can introduce blur, glare, perspective distortion, noise, and changes in screen appearance.
We also discovered that high accuracy on isolated digit images does not automatically translate to reliable end-to-end monitor reading. Screen detection, region localization, segmentation, OCR, and validation all have to work together.
This pushed us toward a more safety-conscious architecture where uncertain readings are surfaced for human review rather than being silently recorded.
Accomplishments that we're proud of
- Built a complete camera → computer vision → OCR → validation → digital record pipeline.
- Supports extraction of six key anaesthesia vitals: HR, SpO₂, NIBP, EtCO₂, temperature, and RR.
- Added an optional ONNX CNN recognition engine alongside traditional Tesseract OCR.
- Built real-time visualization with alerts and trend monitoring.
- Implemented review, correction, sign-off, and append-only audit history.
- Added automatic PDF record generation.
- Designed the system to operate fully offline, without sending patient data to external services.
- Built a calibration workflow so the system can adapt to the physical monitor setup.
What we learned
We learned that building reliable healthcare software is not just about maximizing model accuracy. The entire pipeline and the way uncertainty is handled are equally important.
A model can produce a confident-looking prediction while the upstream localization step has already failed. This made confidence, validation, traceability, and human oversight central parts of our design rather than afterthoughts.
We also learned how challenging computer vision becomes when moving from controlled synthetic images to real-world camera conditions.
What's next for Vital
Our next step is improving generalization across different anaesthesia monitors, manufacturers, screen layouts, and lighting conditions.
We plan to move beyond fixed colour-based localization toward more colour-agnostic visual detection, while continuing to improve robustness under glare, blur, perspective distortion, and partial occlusion.
Ultimately, we want VITAL to become a practical offline-first documentation assistant that reduces manual charting workload while keeping the clinician firmly in control of the final medical record.
Log in or sign up for Devpost to join the conversation.