Inspiration
India's government and semi-private hospitals are critically understaffed. Patients have died waiting in queues because no one was tracking the queue. Medicines have run out mid-treatment because someone forgot to reorder. Follow-up appointments go missed because no one has time to make reminder calls. These aren't edge cases they are daily realities in the hospitals that serve the majority of India's population.
The cruel irony is that AI-assisted hospital management already exists but only in premium private hospitals with large IT budgets and cloud infrastructure. The hospitals that need it most can't afford it, and can't trust it: patient records and drug data cannot go to a third-party cloud under healthcare privacy norms.
That gap is what PaeonOS is built to close.
What It Does
PaeonOS is a local-first AI operating system for understaffed hospitals. It runs entirely on a single laptop inside the hospital — no cloud subscription, no patient data leaving the building — and handles four critical failure points:
- Inbound Voice Scheduling — A VAPI-powered voice agent answers patient calls, understands their request, checks doctor availability, and books or confirms appointments automatically.
- Patient History Retrieval — Doctors speak a patient's name or symptom into their laptop mic. Moss retrieves the full medical record semantically in under 10ms, and reads it back aloud — hands-free, no typing required.
- Pharmacy Inventory Tracking — Every dispense action decrements the live stock count. When any drug falls below 10 units, an alert fires instantly on the staff dashboard.
- Role-Isolated Staff Dashboard — Admins see system health, doctors see patient records and voice query, nurses see the live schedule and booking form. Each role sees only what they need.
How We Built It
The stack was chosen entirely around one constraint: everything must run on the hospital's own hardware, with no patient data ever transmitted externally.
- Moss Python SDK with
SessionIndex— builds and queries the patient index entirely in-process.push_index()is never called, so document text never reaches Moss Cloud. We verified this empirically with a Fiddler network traffic capture — no patient text appeared in any outgoing request body. - FastAPI — the central orchestration layer. Every voice transcript, whether from VAPI or the browser's Web Speech API, passes through an intent classifier that routes it to the right handler: Moss query, schedule lookup, or inventory check.
- VAPI — handles inbound patient calls, speech-to-text, and text-to-speech, with a WebSocket connection to our backend for real-time processing.
- Web Speech API — for the doctor's internal voice query interface, so medical staff can query patient records hands-free without picking up a phone.
- Next.js + Tailwind CSS — the staff dashboard, deployed on Vercel, with role-based routing so each staff type lands on the right view.
- In-memory structured stores — schedule and inventory data live in Python dicts in the backend process. No external database, no ORM, no infra to provision.
# The entire patient index lives here — never pushed to the cloud
_session = await _client.session(index_name="paeanos-patients")
await _session.add_docs(docs)
# push_index() intentionally never called
Challenges We Faced
Memory limits on deployment. Moss's WASM runtime plus the embedding model exceeded Render's free tier 512MB RAM limit. We moved to Railway, which gave us enough headroom to run the full stack.
VAPI's free tier doesn't include ConversationRelay or outbound calls. We had to redesign the voice flow around VAPI's standard tool-call webhook pattern, handling multiple message formats (toolCallList, toolCalls, and legacy functionCall) since VAPI's format varies by account tier and region.
Verifying the privacy claim. "Patient data never leaves the process" is easy to say and hard to prove. We ran the backend through Fiddler Classic with HTTPS decryption enabled, planted a unique marker string (XYZQRT_SECRET_MARKER_12345) in test documents, and confirmed it appeared in zero outgoing requests during add_docs() and query() calls. That result — a tested claim, not a documented assumption — is what makes the privacy story defensible.
Patient ID mismatches. When we expanded from 10 to 1000 synthetic patients, the schedule data still referenced the old 3-digit IDs (P001–P010) while the new dataset used 4-digit IDs (P0001–P1000). A quick Python script remapped the schedule references before anyone noticed in the UI.
What We Learned
- Moss's SessionIndex is genuinely local. Not "local with a cloud fallback" actually local. The network capture proved it. That distinction matters enormously when your pitch is built around data never leaving the hospital.
- Semantic search vs keyword search is best demonstrated with result quality, not just speed. On a deployed server, both modes show ~230ms because network latency dominates. The real proof is asking "elderly patient with breathing problems" and watching Moss find the asthma patient while keyword search finds nothing — same query, meaningfully different results.
- Scope discipline is the real hackathon skill. We started with an Electron desktop app, pivoted to a browser-based voice assistant, redesigned around a voice booking agent for restaurants, and finally landed on a hospital OS. Each pivot happened for a real reason, but the discipline to stop pivoting and start building is what actually got us to a deployed product.
Built With
- moss
- python
- vercel