Inspiration
Our inspiration for this comes from something we hear all the time: the medical field is chronically understaffed, and in crisis or disaster scenarios, that shortage becomes life-threatening. Simultaneously, we believe AI is massively underutilized in healthcare. We connected the dots and thought about the possibility of equipping untrained individuals with AI to allow them to act as effective first responders with nothing more than a simple, portable device? TriageAI was born from the idea of putting an offline, AI-powered triage tool in anyone's hands so that when disaster strikes and every second counts, more people can help even without medical training.
What it does
TriageAI is a portable, offline triage device built on a Raspberry Pi. A first responder points the camera at a wound, presses a button, and the on-device vision-language model assesses the injury's severity (0–3), provides reasoning, and flags cases for escalation. All of this happens without internet access! The responder can then ask follow-up questions via chat, and the device also reads body temperature through an attached sensor (with plenty of room for expansion with other types of sensors). Every assessment is logged locally for later review or for syncing to an online database/dashboard.
How we built it
We started with an architecture diagram and a hardware wishlist: Raspberry Pi, the AI Hat+ 2, a camera module, a temperature sensor, and a portable power supply. After confirming compatibility, we created issues and split into parallel workstreams: backend (FastAPI + SQLite + inference pipeline), frontend (vanilla JS kiosk UI), and hardware (wiring, drivers, Pi setup). The inference layer abstracts over the Hailo NPU running a Qwen2-VL model, with a mock bridge so the software team could develop and test without the physical hardware.
Challenges we ran into
Hardware was overwhelmingly brutal. Driver issues and unforeseen incompatibilities ate up countless hours. The temperature sensor fought us to the point that we needed to set it aside in favor of more important features, our portable charger decided midway through our development process it couldn't supply enough power despite working initially, and the Pi decided to completely stop cooperating on occasion without any interference on our part. On the software side, the AI model only runs on the Pi's Hailo NPU, making it impossible to test inference on a laptop (with the exact same model). We built extensive mocking layers to keep development moving, but integration was still painful. However, the biggest of our challenges was the hardware bottleneck itself: low RAM, a low-res camera, and a small model meant classification accuracy was noticeably weaker than what we saw running a sister model when testing on a MacBook. What we have is a working bare-minimum prototype; it's functional, but constrained.
Accomplishments that we're proud of
Honestly, the biggest win for us is that we managed to produce any kind of MVP at all. Getting the Raspberry Pi fully set up, having our software deployed on it, and setting up an offline VLM actually running inference on-device feels a little unreal to us to have achieved in 24 hours. The whole pipeline, from capture, to assessment, to logging, runs end-to-end on a portable device with no internet. That's something we're really proud of (and felt was important for what we were trying to build here).
What we learned
We learned to think even harder than we did about bottlenecks and constraints before committing to a hardware path. We also dove deep into the differences between LLMs, VLMs, and pure vision models, and the tradeoffs between training your own model vs. leveraging a pre-existing one. We also learned firsthand how much influence context can have on VLM performance. We found that small changes in the assessment criteria had outsized effects on output quality.
What's next for TriageAI
- Better hardware - a more powerful board with more RAM would unlock larger, more accurate models and higher-res imaging.
- More sensors - a pulse oximeter, blood pressure, etc. would give the model richer input for better assessments.
- Online escalation - if internet connectivity is available, we would want to have an option to escalate to a stronger cloud-hosted model (we were heavily eyeing the AMD cloud credits) for a second opinion.
- Doctor-in-the-loop - route escalated cases directly to a remote physician for real-time guidance.
- Multilingual voice input - what we found most exciting was our exploration of 11Labs for real-time speech transcription. This could allow disaster victims to describe their situation verbally in any language, giving the model critical context it can't get from an image alone.
We want give a graphic content warning for the skit video.
Log in or sign up for Devpost to join the conversation.