The home screen with the clock
Adding a medication through the camera
Adding a medication through speech
The 2 options you have to add a medication
All of the medications you have in the app
I was inspired to create this after seeing my grandmother go through the consequence of not taking her pills. This led to critical circumstances and I didn't want this to happen to anyone else. This is why I wanted to create the easiest way to remember to take your pills.
What it does
When you open up the app, it shows you an analog clock which originally has no medications on it.
Then you click the + button and it takes you to a choice of whether you want to take a photo of your pill label or dictate it. If you take a photo, you just take a photo and nothing more. If you dictate, you just read the name and the instructions for taking the pill, then you are done.
Then this adds small circles on the clock that represent when you need to take the pill. Each pill has a different color and this makes it very easy to know when to take the pills.
It also sends you a notification when you need to take the pills.
How I built it
I built it using the Google Cloud Vision API for the OCR(Optical Character Recognition). Then to parse the data, I use the natural language classifier to understand what the doctor is saying in their message on the label. I use the IBM Watson Speech to Text API to determine what the user says when they dictate the message. Lastly, I use the IBM Watson Text to Speech API to dictate back what the program reads, so that people who are visually impaired can listen to the instructions.
Challenges I ran into
- Learning Swift
- This was my very first app in Swift! Learning the language took most of my time but then I got a hang of it and how it was similar to other languages.
- Integrating Google Vision API
- API call was a bit complicated but then I was able to extract the correct information
- Watson Integration
- At the start it was a bit confusing, but once I read into it, I got the hang of it and it was easy from there.
Accomplishments that I'm proud of
I am most proud of the fact that this was my very first app in Swift! This was also the first time that I used IBM Watson and I really liked it, so I think I am going to continue to use it. Lastly, this was my first time using the Google Cloud Vision API, and even though it was hard and took a lot of time, it works out really well and I am really proud of it.
What I learned
- First full app in Swift including UI design
- IBM Watson
- Speech to Text
- Natual Language Classifier
- Text to Speech
- Google Cloud Vision API
What's next for Auxilium Health
- Panoramic Stitching for circular bottles
- Integrations with pharmacies so you can order the pills once you run low
- Improved reliability with OCR and Speech to Text