Arogya-Ai

Arogya AI: An empathetic Bengali AI assistant for instant symptom analysis, prescription decoding, medical report breakdown, and text-to-speech integration.

Comment

Inspiration

In developing countries like Bangladesh, visiting a doctor or understanding complex medical documents can be overwhelming and expensive. Patients often struggle to read handwritten prescriptions, interpret medical reports, or access immediate, reliable advice for common symptoms in their native Bengali language.

We were inspired to build Arogya-Ai to bridge this gap. Our goal was to create an empathetic, AI-powered healthcare companion that simplifies medical jargon, scans physical prescriptions, and offers direct Bengali voice interaction—making healthcare accessible and stress-free for everyone.

What it does

Arogya-Ai is an intuitive web-based health assistant tailored for Bengali-speaking users. It features three primary modules:

  1. Symptom Analyzer : Users can input their symptoms (via text or Bengali voice) to receive instant home remedies, over-the-counter medicine suggestions, and estimated costs in a supportive, empathetic tone.
  2. Prescription Scanner : Users can upload an image of a prescription. The AI decodes the handwritten text, extracts the medicine names, translates dosage instructions into simplified Bengali, and provides estimated prices.
  3. Report Expert : Users can upload medical test reports (like blood tests or pathology reports) to get an easy-to-understand breakdown of the metrics in plain Bengali, helping to ease anxiety before visiting a physician.

How we built it

We built Arogya-Ai using a modern, fast, and robust stack:

  • Backend: Built with FastAPI for high-performance and asynchronous request handling.
  • Frontend: Developed using clean HTML5, Tailwind CSS, and JavaScript for a responsive and intuitive UI/UX.
  • AI Model: Powered by Google Gemini 2.5 Flash (and Gemini 2.0 Flash as a fallback) to achieve lightning-fast multimodal processing (handling text, speech, and images seamlessly).
  • API Integration: Implemented the official google-genai Python SDK with an automated multi-API-key rotation mechanism to handle high traffic and prevent rate limits.
  • Web Speech API: Leveraged native browser capabilities to provide high-quality Bengali voice input and seamless text-to-speech output.

Challenges we ran into

  • Multimodal Image Parsing: Getting accurate text recognition from low-quality, dimly lit images of handwritten prescriptions was challenging. We solved this by refining our prompt engineering and feeding highly structured markdown instructions to the Gemini model.
  • Web Speech Constraints: Handling continuous Bengali voice recognition across different browsers presented stability issues. We tackled this by updating our speech recognition handlers and adding custom fallback alerts for microphone permissions.
  • API Rate Limits: In a live hackathon environment, rate limits can break a demo. We overcame this by implementing a clever backend key-rotation system that dynamically switches between multiple Gemini API keys.

Accomplishments that we're proud of

  • Successfully built a fully functional Multimodal AI application in Bengali within a very tight deadline.
  • Integrated a highly natural Bengali Text-to-Speech (TTS) and Speech-to-Text (STT) system that makes the platform accessible to elderly or visually impaired users.
  • Achieved flawless local and cloud-ready setup using a robust Python backend that processes images in fractions of a second.

What we learned

  • We deepened our knowledge of FastAPI development and dynamic directory configurations.
  • We mastered Advanced Prompt Engineering, especially in instructing LLMs to output strictly structured JSON/Markdown from complex medical images.
  • We learned how to handle asynchronous operations in JavaScript to manage UI loaders, voice control (play/pause/stop), and API requests concurrently without freezing the client.

What's next for Arogya-Ai

  • Local Pharmacy Integration: Partnering with local pharmacies to allow users to order analyzed prescriptions directly through the app.
  • Offline Processing: Exploring lightweight, on-device AI models to enable basic symptom analysis even with poor internet connectivity.
  • Doctor Appointment Booking: Integrating a direct scheduling portal connecting users to real-life specialized doctors when symptoms indicate critical conditions.

Built With

Share this project:

Updates

Submission history