MediScan AI - Powered by Amazon Nova

• 

• 

Inspiration

Honestly, the idea came from something really personal. We have both seen family members come home with medical reports and just stare at them, completely lost. A printed sheet full of numbers, arrows, and terms like HbA1c, serum creatinine, TSH — and nobody in the house knows what any of it means. Usually the paper just gets put aside until the next doctor visit, while the person spends days worried and uncertain. That bothered us. Health information belongs to the patient. They should be able to understand it without needing a medical degree or waiting a week for an appointment. When we found out that Amazon Nova 2 Lite is a multimodal model -meaning it can actually read a PDF or image the way a person would , we knew immediately that this was the technology that could make our idea real. No other approach would work. You cannot solve this problem with a text-only model. You need something that can look at a document and understand it.

That is where MediScan AI started.

What it does

MediScan AI lets you upload any medical report — a lab result, a blood test, a prescription, even an X-ray image — and it reads the whole thing automatically. No typing, no copy-pasting. You just drop the file and the app does the rest. It pulls out all the lab values, checks which ones are outside the normal range, identifies any medications, and gives you a Health Score out of 100. Then it writes a short summary in plain, simple language so anyone can understand what their report is saying — without needing a medical background. After that you can ask questions. Type something like "should I be worried about my glucose?" or just click the Speak button and ask out loud. The app answers based on your actual report, not some generic internet response. Amazon Nova 2 Sonic handles the voice side — it listens, thinks, and speaks the answer back to you in real time. There is also a History tab that keeps track of your past reports and shows you trends over time, so you can see if something like

your cholesterol or blood sugar has been slowly changing.

How we built it

We split the work between us from the start. Hamna built the entire frontend — the interface, the upload box, the chat area, the voice button, the results display, and the History and About pages. We kept it as a single HTML file so it opens in any browser without needing to install anything.

Shumaila built the entire backend using Python and FastAPI. This is the part that connects to AWS Bedrock and talks to the Amazon Nova models. When a file gets uploaded, the backend converts it to Base64 format and sends it to Amazon Nova 2 Lite, which reads the document directly as a multimodal input. The results come back as structured JSON and get displayed on screen.

For chat, the backend keeps the full conversation history and sends it with every message so Nova always knows the context. For voice, it uses bidirectional streaming with Nova 2 Sonic — audio goes in, audio comes back out, all in real time.

We also added an agentic layer where Nova can use tools on its own when it needs to — like looking up normal reference ranges, checking for drug interactions, or calculating a risk score. Nova decides which tools to call, uses them one by one, and puts together a final answer from the results.

For document search across multiple uploads, we used Nova Multimodal Embeddings. Each document gets converted into a mathematical vector, and when you ask a question, the app finds

the most relevant document using cosine similarity.

Challenges we ran into

The biggest technical challenge was Nova 2 Sonic. It uses bidirectional streaming which is very different from a regular API call. Audio has to go in and come back out at the same time, like a live phone call. Getting that stream set up correctly — the session start, the audio chunks, the response events — took us a lot of time and testing to figure out.

Sending files to a multimodal model was also something we had not done before. We had to learn how to convert files to Base64 and structure the request exactly the way Nova expects it. The format is specific and if anything is wrong, the model simply does not respond the way you want.

Writing the medical prompts carefully was another challenge we took seriously. Health is a sensitive area. We did not want the app to say something incorrect or alarming. So we spent a lot of time writing and adjusting the system prompts to make sure Nova is always helpful, always accurate, and always tells the user to consult a real doctor for final decisions. On the design side, we wanted the app to feel like a real product and not a rough student project. Getting the layout, the color coding, and the interactions to feel natural took more effort

than we expected.

Accomplishments that we're proud of

Honestly, the thing we are most proud of is that the app works the way we imagined it from the beginning. The idea of uploading a real medical document and having AI read it and explain it — that actually works, and it works well.

We are proud that we used three Amazon Nova models together in one project, each doing a different job. Nova 2 Lite for reading documents and chat, Nova 2 Sonic for voice, and Nova Embeddings for smart document search. Getting all three working together in a single app felt like a real achievement.

We are also proud of the voice feature. The fact that an elderly person with no technical skills can just click one button, speak a question, and hear the answer back — that felt meaningful to us. It was not the easiest thing to build but it was worth it.

And we are proud that we built this as a team, divided the work clearly, and finished it before the deadline. This was the most

complex project either of us has worked on so far.

What we learned

We both learned a lot of things we had never done before.

On the technical side — we learned how multimodal AI models work and how to send images and PDFs as inputs. We learned how AWS Bedrock works and how to connect to it with boto3. We learned how bidirectional audio streaming works with Nova 2 Sonic. We learned how to build and run a FastAPI backend. We learned how embeddings work and how to use cosine similarity for search.

We also learned how much prompt engineering matters. Especially in a sensitive domain like healthcare, the way you write your system prompt completely changes how the model behaves. We rewrote our prompts several times before we were happy with them.

But the biggest thing we learned was about building for real people. Every decision — the voice button, the plain-language summaries, the quick action buttons — came from thinking about someone who is not technical, who is worried about their health, and who just needs a simple answer. That way of thinking changed

how we approached every part of the project.

What's next for MediScan AI - Powered by Amazon Nova

We have several ideas for what we would add if we continue working on this project.

The first thing we want to add is Urdu language support. Most of our potential users in Pakistan are more comfortable in Urdu than English. Adding Urdu text and voice responses would make the app useful for a much larger audience.

We also want to add a doctor sharing feature — where after the AI analysis, the patient can send a clean summary directly to their doctor through WhatsApp or email. This would make the doctor's appointment faster and more productive.

Medication reminders are another thing we want to build. Once the app extracts medications from a prescription, it should be able to set daily reminders so the patient never misses a dose.

We also want to handle mixed-language reports better. In Pakistan, many reports are partly printed in English and partly handwritten in Urdu or with local abbreviations. Training the model to handle these would make it much more practical.

And long term, we would love to integrate with health wearables so readings like blood pressure and blood sugar come in automatically without needing a separate upload every time.

The goal stays the same though — make health information accessible to every patient, regardless of their education, location, or language.

Shumaila Tariq & Hamna Maqbool | #AmazonNova

Built With

• amazon-nova
• amazon-nova-2-lite
• amazon-nova-2-sonic
• amazon-nova-multimodal-embeddings
• aws-bedrock
• base64
• boto3
• cosine-similarity
• css3
• fastapi
• html5
• javascript
• json
• multimodal-ai
• python
• rest-api
• uvicorn
• voice-ai
• web-speech-api