🚑 The First Respondent — Project Story
🌟 Inspiration
According to the WHO, 2.2 billion people globally have vision impairment, and millions more find themselves in remote areas with no cell service. When you're deep in the wilderness, during a natural disaster, or facing a medical emergency at home, cloud-based AI and telehealth apps are useless without connectivity. Traditional first-aid manuals are static, require manual searching, and have zero situational awareness—they don't know what they're looking at, and they can't talk you through it.
The First Respondent was inspired by a simple question: What if your phone could see the injury and guide you through exactly what to do—all without any internet connection?
I witnessed my grandmother struggle with vision loss, frequently asking "What color is this pill?" or "Can you read this label for me?" While voice assistants exist, they can't see. Commercial AI vision services require sending images to the cloud, which raises privacy concerns and becomes impossible during internet outages.
This project was born from a mission: What if everyone could have a private, always-available AI assistant that could see for them, without relying on the cloud?
💡 What It Does
The First Respondent is a fully offline Progressive Web App (PWA) that provides AI-powered first aid, survival, and emergency guidance. It works entirely in your browser with no internet connection required after initial setup.
Key Features:
- Offline AI Chat: Uses Phi-3.5-mini (2GB model) running via WebLLM to provide step-by-step emergency guidance
- Camera + Vision Analysis: Captures photos and analyzes injuries using ResNet-50 via Transformers.js
- Curated Knowledge Base: 40+ vetted first-aid and survival entries for grounded, accurate responses
- True Offline Operation: Service worker caches everything — works in airplane mode
- Privacy by Design: No data ever leaves the device — no cloud APIs, no servers
- Safety First: Persistent safety banner and prompt engineering prevent dangerous advice
How It Works:
- First Load: Downloads the AI model (~2GB) with progress bar
- Chat: Type any emergency question — AI responds with step-by-step guidance
- Camera: Tap 📸 to analyze wounds, burns, or injuries
- Offline: After first load, works completely without internet
Example Responses:
| User Query | AI Response |
|---|---|
| "I cut my hand" | "Apply direct pressure with a clean cloth. Elevate the wound. If bleeding doesn't stop in 10 minutes, seek medical help." |
| "How do I treat a burn?" | "Cool the burn under running cool water for 10-15 minutes. Do not apply ice. Cover with sterile dressing." |
| "What do I do if I see a bear?" | "Don't run. Back away slowly. Make yourself look big. If attacked, play dead for bears." |
| "How do I purify water?" | "Boil water for 1 minute (3 minutes above 2000m). Use purification tablets as alternative." |
🛠️ How We Built It
Technology Stack
┌─────────────────────────────────────────────────────────────┐
│ Frontend (React + Vite) │
│ ┌────────────┐ ┌────────────┐ ┌──────────────────────┐ │
│ │ Chat UI │ │ Camera │ │ Safety Banner │ │
│ │ (React) │ │ (WebRTC) │ │ (Persistent) │ │
│ └────────────┘ └────────────┘ └──────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
│
┌───────▼───────┐
│ Service │
│ Worker │
│ (Offline) │
└───────────────┘
│
┌─────────────────────────────────────────────────────────────┐
│ AI Models (100% Client-Side) │
│ ┌──────────────┐ ┌──────────────┐ ┌────────────────────┐ │
│ │ WebLLM │ │Transformers.js│ │ Knowledge Base │ │
│ │ Phi-3.5 │ │ ResNet-50 │ │ (JSON) │ │
│ │ (2GB) │ │ (Image) │ │ (40+ entries) │ │
│ └──────────────┘ └──────────────┘ └────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
Key Technical Decisions
1. Model Selection
- Phi-3.5-mini (2GB): Chosen for its balance of capability and size — runs in browser via WebLLM with 512 token context, temperature 0.3 for factual responses
- ResNet-50 (100MB): Image classification via Transformers.js for injury analysis
- Knowledge Base: Curated JSON of 40+ first-aid and survival entries for grounding
2. Architecture Decisions
- Vite + React: Lightning-fast builds and hot reload during development
- WebGPU: Near-native inference performance in Chrome/Edge 113+
- Service Worker: Caches everything for true offline operation
- Progressive Web App: Installable on mobile devices
3. Prompt Engineering
You are The First Respondent, an offline emergency assistant.
Provide practical, step-by-step guidance for first aid, survival, and emergencies.
Use this verified knowledge when relevant:
[Retrieved Knowledge Base Entries]
Guidelines:
- Be specific with steps
- Prioritize life-saving actions
- Say "I don't know" if unsure
- Include safety warnings
- NEVER provide medication dosages
🧩 Challenges We Ran Into
1. Dependency Hell
Problem: @mlc-ai/web-llm, @xenova/transformers, and workbox-window had conflicting peer dependencies, causing npm install failures.
Solution: Used exact versions and --legacy-peer-deps flag:
"dependencies": {
"@mlc-ai/web-llm": "0.2.46",
"@xenova/transformers": "2.17.0",
"workbox-window": "7.0.0"
}
2. Cross-Origin Headers
Problem: WebLLM requires Cross-Origin-Opener-Policy and Cross-Origin-Embedder-Policy headers to work in the browser, otherwise it fails to load.
Solution: Added headers in vite.config.js:
server: {
headers: {
'Cross-Origin-Opener-Policy': 'same-origin',
'Cross-Origin-Embedder-Policy': 'require-corp'
}
}
3. Large Model Downloads
Problem: Phi-3.5-mini is ~2GB — first load is extremely slow and users may think the app is broken.
Solution: Added clear progress bar and messaging:
- "Downloading AI Model..." with percentage
- "~2GB - First load only. Cached for offline use."
- "Keep this tab open and stay on Wi-Fi"
4. WebGPU Compatibility
Problem: Not all browsers support WebGPU, causing the app to fail silently.
Solution: Added graceful fallback:
if (!navigator.gpu) {
setWebgpuSupported(false);
setModelReady(true); // Use fallback responses
}
5. Service Worker Caching Limits
Problem: The ~5MB JavaScript bundle was too large for service worker pre-caching (limit is ~2MB).
Solution: Configured Vite PWA to handle large files via browser cache instead of service worker cache.
6. Model Loading Time
Problem: First inference after loading the model takes 10-15 seconds on CPU-only devices.
Solution: Added loading state and "Thinking..." indicator so users know the app is working.
🏆 Accomplishments We're Proud Of
1. True Offline Operation
We built a fully functional AI assistant that works in airplane mode. No cloud APIs, no internet connection needed. This is the single most impressive feature for our pitch.
2. Sub-2GB AI Model
We successfully run a 2GB LLM (Phi-3.5) entirely in the browser. Most people don't think this is possible, but WebGPU and WebLLM made it happen.
3. Camera + AI Integration
We combined computer vision (ResNet-50) with the LLM to analyze injuries and provide first aid guidance — all client-side.
4. Curated Knowledge Base
We built a database of 40+ vetted first-aid and survival entries, ensuring the AI gives grounded, accurate responses instead of hallucinations.
5. Privacy by Design
No data ever leaves the user's device. This is a major competitive advantage over cloud-based alternatives.
6. Safety First
We designed the entire system with safety in mind — persistent safety banner, prompt engineering, and refusal to give dangerous advice.
7. Installable PWA
Users can install the app on their phone's home screen, making it feel like a native app.
8. Accessibility Design
Large buttons, clear messaging, and a clean
Built With
- css-frameworks:-react
- github-pages-testing:-chrome-devtools
- html
- languages:-javascript
- onnx-runtime-deployment:-netlify
- service
- transformers.js
- typescript
- vercel
- vite-ai/ml:-webllm
- worker
Log in or sign up for Devpost to join the conversation.