Insight-AI

AI generated key insights, recommendations and infographic image after processing deceased clinical note/ symptoms that is uploaded as a .pdf or an image document.

Comment

Inspiration

Health clinics are sometimes overwhelming with patients and the doctors are busy. Providing individual insights, recommendations, and care can take time and effort that doctors may not usually have time for. The deceased and their friends/relations have to wait for days and sometimes hours. Hence, "Insight-AI" is designed not only to provide key insights and recommendations for a patient, but also provide an infographic image summarizing the content. This can reduce the time spent in the doctor's office as it can help doctor and the patient. The patient and their relatives/friends will have a guided recommendation that they can show to the doctor for approval/changes, thus minimizing time, and effort.

What it does

The project takes in the pdf or image file that is a patient's clinical note or symptoms and shows the content in one text area. Upon clicking the "Key Insights" button, the application provides key recommendations and insights for the patient and also generates an infographic image.

How it was built

The project is built in Java Script, AWS Lambda, Hugging Face Inference provider, Amplify and Vite. The models used from Hugging Face inference is Deepseek-ai's R1-0528 model and stable diffusion AI's '3.5-large' model. The front end is designed using Type Script / Java Script. Vite is used to create and package Java Script application, AWS Lambda is used to get the API keys from Hugging Face tokens and encrypt it. The Lambda API Gateway transmits the encrypted keys and the keys are decrypted in the client-side. Once the keys are decrypted it is validated and key insights and recommendations are provided. Stable diffusion's '3.5-large' model' from Hugging Face l is used to generate the infographic image.

Challenges

  1. Research on the right model to use which can accept large texts as input and generate a detailed recommendation.
  2. Extract text from the .pdf and image document.
  3. To find the right diffusion model to generate a relatable image.
  4. To synchronize the workflow so that the document shows the key insights and recommendations first, and then generate an image.
  5. Identify the right parameters in the stable diffusion model to try and generate a relatable image.
  6. Engineering prompt to provide relatable results based on the type of document.
  7. Clear mention of Usage and terms.

What we learned

Learned about the noising and de-noising to create realistic images in Stable diffusion model. Also, learned about the thermodynamics science principle used to create various AI model's. Powerful feature of encoding and decoding in Amazon Stable diffusion as compared with Open AI encoder only model.

What's next for "Insight-AI"

The infographic image generated doesn't have proper readable, totally relatable text even with attempted "few shot prompting". Hence, need to improve on generating a more valid, relatable infographic image.

Built With

Share this project:

Updates