Team MedScribe

Seamless Consultation, Effortless Reports

Comment 3

MedScribe

Inspiration

We set out to make doctor-patient interactions in telehealth smoother and more efficient. One of the biggest challenges we saw was the difficulty of capturing and documenting important details from online consultations. To solve this, we designed an application that not only enables seamless video communication but also automates report generation. Today, we've turned our idea into reality with a Minimum Viable Product (MVP). Our MVP allows doctors to have live video consultations, generates real-time transcripts, and creates automated reports. This is a big step toward our goal of simplifying medical documentation, making doctors’ work easier, and improving the overall patient experience.

What It Does

MedScribe is a unified telehealth application that:

  • Enables real-time doctor–patient video consultations and chat, ensuring seamless communication.
  • Integrates AI-assisted report generation: After each consultation, our system processes transcripts using LLMs (and Gemini API) to produce editable, accurate medical consultation records.
  • Provides a universal datastore: All patient records, including imaging and test results, are securely stored in one place. This ensures that historical data is always accessible and minimizes the risk of misplaced medical reports.

How We Built It

Tech Stack & Architecture

  • Front End: ReactJS, CSS, and TailwindCSS for a responsive, modern user interface.
  • Back End: Flask and Python powered the REST API endpoints and handled real‑time communication via Socket.IO and WebRTC (with PeerJS) for video calls.
  • AI & Data Processing: We integrated AWS Transcribe for converting consultation audio to text, and then passed the transcripts to the Gemini API for automated report generation.
  • Storage & Security: Patient records and reports are stored securely using AWS S3, while AWS RDS and IAM ensure data integrity and access control.
  • Deployment: The solution is hosted on AWS EC2, with various AWS services orchestrating data storage and processing.

Challenges We Ran Into

Some of the key challenges included:

  • Implemented real-time video and chat communication and generating faster reports seamlessly in one platform.
  • Creating a universal storage system to consolidate and secure diverse patient data.
  • Adopting latest technologies and trends (AWS services, generative AI, WebRTC) into a cohesive system.
  • Developing AI workflows that produce reliable and editable consultation report summaries.

Accomplishments That We're Proud Of

  • Delivered a working MVP that brings together real-time video consultations, chat, and AI-driven report generation.
  • Developed a universal datastore that securely holds all patient records, overcoming the issue of data silos.
  • Integrated cutting-edge technologies such as AWS Transcribe, generative AI (via Gemini API), and WebRTC for real-time communication—all within a short timeframe.
  • Validated our concept by addressing real-world challenges in clinical communication and decision-making.

What We Learned

Our journey taught us that:

  • Interdisciplinary Collaboration is Key: Bringing together expertise in software development, cloud services, and AI is crucial for tackling complex healthcare challenges.
  • Building an MVP and refining based on feedback is vital, especially in hackathon settings.
  • Integration of Emerging Technologies: We learned how to combine AI, real-time communications, and secure cloud storage to create a seamless user experience.
  • The Importance of Security and Compliance: Handling sensitive healthcare data requires strict adherence to security standards and continuous evaluation of our data protection strategies.

What's Next for Team MedScribe

Looking ahead, our roadmap includes:

  • Security - Encryption of Data and storing safe, Controlling the Data that is passed to LLM, Hosting own LLM Model, Restricting the API - Bearer Token etc.
  • Application - Making better UI, Creation of Full Appointment Booking Management, Enhanced Video and Chat Communication, Better Deployment
  • GenAI Usecases - Personalized Chatbot, More Enhanced Visual Dashboards, More Categories in Reports Generated, Synthetic Data Generation, Apple Health Kit Data Integration, MultiLingual Reports, Reducing Human Errors
  • Optimizing our architecture for scalability and ensuring that our solution can support a growing user base in a real-world clinical setting.

Built With

  • amazon-web-services
  • aws-sdk-(boto3)-aws-services:-ec2-for-deployment-rds-for-database-and-iam-aws-transcribe-s3-for-storing-health-records-(blob-storage)-api:-gemini-api-for-llm-analysis-of-transcripts-and-reports-(generative-ai)
  • boto3
  • css
  • css3
  • ec2
  • gemini
  • google-generativeai
  • iam
  • mongodb
  • ngrok
  • postman
  • python
  • rds
  • react
  • rest
  • rest-api-endpoints
  • s3
  • services
  • tailwindcss-back-end:-flask
  • transcribe
  • webrtc
+ 5 more
Share this project:

Updates