vaidya AI

"Listens to the doctor. Writes for them." — simple, literal, memorable

Comment

Vaidya AI — listens to the doctor, writes for them

Inspiration

The spark for this came from a single statistic that wouldn't leave us alone: doctors in India spend roughly 3.4 hours a day on paperwork — writing notes by hand, filling prescriptions, drafting referrals — time that should belong to patients, not forms.

At the same time, India's Ayushman Bharat Digital Mission (ABDM) is pushing every hospital above 100 beds toward digital, standards-compliant health records by 2025–26. That's a deadline, not a suggestion — and most hospitals have no easy way to get there.

Every ambient documentation tool we looked at (Nuance DAX, Suki, Abridge) was built for English-only, US-priced, US-workflow healthcare. None of them understood Hindi-English code-switching, none were priced for the Indian market, and none spoke ABDM's language out of the box. That gap — no India-first, ABDM-native, multilingual scribe exists — is what we set out to fill.

What it does

Vaidya AI sits in the consultation room and listens. As the doctor and patient talk — in English, Hindi, or the code-switched mix that's actually spoken in most Indian clinics — it:

  1. Transcribes the conversation in real time
  2. Structures it into a clinical SOAP note (Subjective, Objective, Assessment, Plan)
  3. Extracts a formatted prescription from anything mentioned
  4. Lets the doctor review and approve the note with one tap — never auto-submits anything unreviewed
  5. Exports the approved record as a FHIR R4 resource, the data standard ABDM requires

The whole loop — from spoken word to a doctor-approved, standards-compliant record — happens in under two minutes per consult.

If a doctor sees $N$ patients a day and Vaidya saves $t$ minutes of documentation per consult, the time recovered is:

$$ T_{saved} = N \times t $$

For a typical OPD doctor seeing 60 patients and saving even 4 minutes per consult:

$$ T_{saved} = 60 \times 4 \text{ min} = 240 \text{ min} \approx 4 \text{ hours/day} $$

That's almost a full extra shift's worth of attention given back to patients, every single day.

How we built it

We gave ourselves one rule for the week: build the thinnest slice that proves the magic moment, not the whole platform.

  • Speech-to-text: Whisper API, chosen because it handles Indian-accented English and Hindi reasonably well out of the box without fine-tuning
  • Note generation: Claude, prompted to convert raw transcripts into structured SOAP notes and prescriptions — this is where most of our actual engineering time went
  • Frontend: Next.js, deployed on Vercel for a live, shareable demo link
  • Storage: Supabase for sample patient records and generated notes
  • ABDM-readiness: rather than integrating the real ABDM sandbox (which requires weeks of approval we didn't have), we built a correctly formatted FHIR R4 JSON export — proving technical fluency with the standard without needing live access

Our sprint broke down roughly like this:

Day Focus
1 Repo, skeleton app, mic capture, audio → placeholder text
2 Real Whisper pipeline, tested against Hindi-English samples
3 SOAP note prompt engineering — the heart of the product
4 Doctor review UI — record, transcript, draft, approve
5 Prescription extraction + mock FHIR R4 export
6 Polish, pre-recorded backup audio for demo-day safety
7 Pitch deck, full demo rehearsal

Challenges we ran into

Code-switching broke our first transcription pipeline. Real Indian consultations don't stay in one language sentence to sentence — they switch mid-thought. Our first pass mangled half of these. We fixed it by testing against real recorded sample conversations early, rather than clean single-language audio, and tuning model settings accordingly.

Getting the LLM to be consistently clinical, not just fluent. Early prompt versions produced text that read like a note but wasn't reliably structured — fields missing, inconsistent formatting. We iterated through more than ten prompt versions, testing against the same set of sample transcripts each time, until the output was dependable enough to trust.

No real ABDM sandbox access in a week. Government API approval takes far longer than a hackathon allows. Rather than skip the integration story entirely, we built a hand-verified FHIR R4 mock export — which forced us to actually learn the standard properly instead of hiding behind a real API call we didn't understand.

AI confidence vs. clinical safety. The scariest realization mid-build: a hallucinated detail in a medical note isn't a cosmetic bug, it's a patient-safety issue. That reframed our whole approach — the doctor-approval step stopped being "a feature" and became the non-negotiable core of the design.

Demo-day risk. Live mics fail. We recorded backup conversation clips in advance so a flaky microphone on stage couldn't sink the whole pitch.

Accomplishments that we're proud of

  • A working end-to-end pipeline: live voice in, doctor-approved structured record out, in real time
  • Genuine Hindi-English code-switch handling, not just a translated demo
  • A FHIR R4 export that's actually standards-correct, not just decorative JSON
  • A doctor-approval step built in by design, not bolted on as an afterthought

What we learned

  • How deep the administrative burden on doctors actually runs, and how much trust has to be earned before anyone hands AI their clinical notes
  • The real shape of FHIR/HL7 and how ABDM's HIP/HIU/consent framework fits together
  • That prompt engineering for high-stakes structured output is its own discipline — fluent isn't the same as correct
  • That in a hackathon, one believable magic moment beats ten half-built features every time

What's next for Vaidya AI

  • Pursue real ABDM sandbox certification rather than the mock export
  • Pilot with a handful of real outpatient departments to test accuracy against actual clinical workflows
  • Expand language coverage beyond Hindi-English to Tamil, Telugu, and Kannada
  • Layer in the next module — billing and claims auto-fill — reusing the same clinical data already being captured, the first step toward a full hospital operating system built one trusted product at a time

Built With

  • and-note-generation-requests-(alternative:-next.js-api-routes-if-you-want-backend-and-frontend-in-one-codebase
  • and-type-safe-queries-against-postgres-docker-?-runs-postgres-locally-during-development/demo
  • apis
  • bcrypt
  • consults
  • express.js
  • generated-notes-prisma-?-orm-for-schema-modeling
  • handles-hindi-english-code-switching-ai-/-note-generation-claude-api-?-transforms-transcripts-into-structured-soap-notes
  • jwt
  • live-transcript
  • migrations
  • next.js
  • no-external-service-dependency-authentication-jwt-(json-web-tokens)-?-custom-auth-for-doctor-login-sessions
  • no-separate-server)-database-(native)-postgresql-?-relational-database-for-patients
  • node.js
  • postgresql
  • prescriptions
  • referrals-frontend-next.js-?-doctor-facing-app-(record-button
  • review/approve-ui)-vercel-?-hosting-and-deployment-for-the-live-demo-link-backend-(native)-node.js-?-runtime-express.js-?-rest-api-server-handling-audio-upload
  • speech-to-text-whisper-api-?-converts-doctor-patient-audio-into-text
  • transcript-processing
Share this project:

Updates

Submission history