write to form

Inspiration

We felt that this was a very important topic to us as one of our parents works as a nurse. In the medical field, upwards of 27% of a medical professional's time is spent on manually inputting handwritten forms and documents into electronic systems. In order to elleviate some of the strain put on our career physicians we wanted to reduce the time spent on paperwork, to let them get back to what matters most, saving valuable time to save lives.

What it does

Our program takes a user selected input image / scan to read handwriting and printed text, it then creates a csv file to match our form template and then autofills the information so it can be received as an excel file and then uploaded into a given medical database according to the information provided.

How we built it

We used Tesseract-OCR to first read the hand-written form submitted by the user character by character, then the text is input to Gemini where it is transformed into a CSV file that can be opened as an Excel file and then the information can be extracted or read by your given medical database.

Challenges we ran into

We ran into several challenges with the formatting of our forms and the handwriting we had used to write on them. If we used cells to order our information on the form, the vertical lines seperating segments were read as I's in our code. We also ran into issues with data reading inaccuracies, where certain characters are read incorrectly completely changing the meaning of the handwritten note.

Accomplishments that we're proud of

Getting our files to load to correctly, as this is our first time using LLM's in code and using AI in our code in general. We're proud to see that our code works as we have never worked with this sort of interfacing before.

What we learned

How to use Gemini (LLM's in general) and that it can incorporated practically into our code to significantly ease our work load. We also learned what an API key is and that we could use it to secure out program. We also learned how to work with an OCR (optical character recognition) in order to read our forms.

What's next for write to form

Next we want to work on improving the accuracy of our writing, correcting spelling errors that are common and visibly wrong (ex. obvious mispellings of common words or perscriptions). We also want to improve on the recognition of common words- for example a Doctor's handwriting would appear multiple times on the app and over time we would hope that the individual's handwriting would be recognized with a higher success rate as the system can train on their handwriting and common writing ticks (the way someone writes their i's or z's).