-
-
Architecture Diagram
Inspiration
I'd already built MAN (Medical Audio Notes) before this hackathon — a side project born out of watching how much time doctors lose to documentation instead of actually looking at their patients, and how badly that gets worse when the patient doesn't speak the doctor's language.
Multilingual clinics are common where I'm from, and "the doctor typing while the patient talks" is a universal complaint.
When the Qwen Cloud Hackathon came up, I saw a chance to push the idea further than a personal side project: not just transcribe-and-summarize, but give the app an actual memory — something that gets smarter about a specific patient the longer you use it, instead of treating every visit like the first one.
What it does
A doctor logs in, adds a patient, and starts a visit one of two ways:
Record Symptoms: The patient speaks naturally — in any of 30+ languages — and Qwen Cloud transcribes and restructures it into a full clinical note:
- Chief complaint
- HPI
- Timeline
- Triage level (🔴 / 🟡 / 🟢)
- On-demand differentials
- Suggested investigations
Speak to Doctor: The patient has a live conversational intake with an AI that asks smart follow-up questions and, on finishing, produces a clinical summary plus an explicitly labeled provisional diagnosis — clearly flagged as AI-generated and not something to act on without seeing a real doctor.
Either way, nothing saves to the patient record until a human clinician reviews and confirms it.
The part I care about most: every patient has a single, compact memory profile that Qwen rewrites after every visit — keeping allergies, chronic conditions, and medications, folding in the new complaint, and actively dropping resolved short-term symptoms once a follow-up comes back "Better."
So the AI's context about a patient stays useful and small instead of growing into an unusable pile of old visit logs.
How I built it
The application uses:
- Flask + SQLite for the backend
- Vanilla JavaScript for the frontend
- Qwen Cloud through Alibaba Cloud Model Studio's OpenAI-compatible endpoint for all AI operations
All LLM calls — including:
- Note generation
- Triage
- Differential diagnoses
- Conversational intake
- Memory summarization
— run through Qwen Cloud, using qwen-max for language tasks and qwen3-asr-flash for the speech-to-text pipeline.
The application is deployed on an Alibaba Cloud ECS instance, using:
- Docker
- Gunicorn
- Nginx reverse proxy
- Free Let's Encrypt SSL certificate
I used an sslip.io hostname since I didn't want to purchase a domain just for a hackathon demo. HTTPS was necessary because browsers block microphone access on plain HTTP for anything that isn't localhost — something I found out the hard way.
Challenges we ran into
Almost every challenge was "the docs say one thing, the API says another."
qwen-audio-asr doesn't actually exist on the compatible-mode endpoint. It took some trial and error against the live API to land on qwen3-asr-flash, and even then the audio payload needed a full:
data:audio/wav;base64,...
URI rather than raw Base64 like I first assumed.
My account's free tier had quota for qwen-max and qwen-flash, but not qwen-plus or qwen-turbo, which only became apparent through a 403 error at request time.
On the infrastructure side, the free-trial ECS wizard handed me a CentOS box instead of the Ubuntu environment I'd planned for, with no SSH key — just a reset-the-password-and-reboot flow.
Getting a fresh RAM user working meant discovering, one permission error at a time, that ECS, OSS, and ACR access doesn't automatically include VPC access — that's a separate policy.
Even with a $40 trial coupon sitting on the account, Alibaba Cloud still requires a bound payment method before it'll allow pay-as-you-go resources to launch.
The challenge that mattered most for correctness was a silent visit-summary parser bug.
My parser expected numbered headings like:
1) Summary
But Qwen generated Markdown headings like:
### Summary
As a result, every section was quietly coming back empty — not just the new provisional-diagnosis field.
Nothing crashed. It just didn't work, which is the worst kind of bug to catch.
I found it by printing and inspecting the raw model output during testing instead of trusting the parsed result.
Accomplishments that we're proud of
- Catching the silent parsing bug before it shipped.
- Getting a real, trusted HTTPS certificate running on a bare IP with no purchased domain, entirely for free.
- Watching the memory-forgetting behavior work end to end.
When a follow-up is marked "Better," the patient's memory profile visibly drops the resolved complaint during the next regeneration while keeping important information such as allergies and medication history intact.
That's the whole thesis of the MemoryAgent track working in front of me, not just in theory.
What we learned
Model availability and payload formats are worth verifying against the live API before writing a single line of integration code, no matter how confident the documentation sounds.
We also learned that "the AI asks the patient for a diagnosis" may look like a small UI copy problem, but it's actually a real safety problem. If there's no clinician in the room, you can't ask the patient to supply medical judgment they don't have.
Finally, a lot of what looks like "cloud deployment friction" is really just RAM/IAM permission modeling. ECS, VPC, OSS, and Container Registry each have their own policies — and that's by design, not an accident.
What's next for Anamnesis
The next steps are:
- Getting OSS fully activated so PDF exports archive automatically.
- Building richer structured memory, with separate fields for chronic conditions and active complaints instead of one free-text blob.
- Adding multi-clinician support for a shared patient record.
- Eventually running a real pilot with an actual multilingual clinic to see whether memory recall changes how a return visit actually goes.
Log in or sign up for Devpost to join the conversation.