Inspiration
In many countries around the world, healthcare institutions are crumbling due to the shortage of doctors, nurses and other healthcare professionals post pandemic. However, many of the people currently filling up these hospitals and ERs may not need to be there; my friend's brother had to wait a few months for a life saving procedure because the hospitals were filled with potential COVID patients at the time. A COVID test or an app such as Doct.AI could have remediated this issue much faster and have made better use of the doctors' time. So at CruzHacks, I decided to enact on the idea!
What it does
When you notice something irregular with your body, you type in some symptoms that you have been experiencing over the past few days. If words can't describe the symptom well, you can also upload an image which the backend will analyze. In either case the app sends back an analysis and possible theory of the disease / issue you may be experiencing and next steps to remediate it. Keep accessibility and UX in mind, users who may not be able to type or are visually impaired can also relay their symptoms via speech through a microphone on the app which will transcribe their words into the prompt box. If the app detects you may require a health professional, it suggests some around the area and can contact them on your behalf to get an appointment / consultation going faster.
How we built it
- Azure Cognitive services for speech -> text
- Salesforce Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation (BLIP) model for image analysis
- Twilio to send texts to nearby doctors / specialists
- Node JS and Python for backend API calls to ML models
- React.js for frontend
- OpenAI's ChatGPT
Challenges we ran into
Some of the libraries I wanted to use were not fully functional with the latest ML models and sometimes couldn't even integrate with the Javascript code anyway. So I had to shell out to Python for some API calls and eventually had to synchronize the calls that the frontend was making with 2 backends.
Accomplishments that we're proud of
Having a decent working demo lol.
What we learned
ChatGPT and other models like Saleforce's blip are so advanced nowadays 🤯. ML is the future!
What's next for Doct.AI
- Integrate a Map API to automatically search for nearby specialists
- Keep track of a user's previous queries so the knowledge base of the AI for the specific user becomes more precise and hopefully more helpful
Log in or sign up for Devpost to join the conversation.