Billify

Inspiration

We wanted to make something to simplify an everyday task, like reading bills. Another reason for going with this is that we all decided that we wanted to work on a multimodel project

What it does

With this application a user can scan any medical bill that they have, to get a simplified version of it that they can easily understand, and it also highlights certain malpractices users are encountering with the bill and how to negotiate against these malpractices. But we added our own twist to it, basically the reason why people do not just enter their medical documents into a chatbot is due to privacy concerns. So, before we send the user data anywhere, we use a local model to redact all personal information.

How we built it

For the redaction process, we used a specialized local AI model capable of generating OCR text from an image, identifying sensitive information, and redacting it. This then gets sent to the Gemini-API so that it can generate a response for the user using the redacted OCR data

Challenges we ran into

We initially tried to do the redaction process with the image itself (no extracting text), but that did not work out because the models we tested were not reliable enough. Then we finalized on our present method for redaction but even that had a few issues with model selection and actually making it work with our data. Other than that, we also ran into some troubles with the frontend because we decided to use React for it and none of us had experience with that, so we had to learn it before implementing it to our project. And because of this we also had issues with connecting our frontend and backend with FastAPI mostly because we didn't exactly plan out our code with this step in mind

Accomplishments that we're proud of

The most obvious one would be the frontend using react. Using this framework made it a lot easier to work on the design process and make it look good compared to just hard coding all the frontend. We are also happy with how our redaction process works because that's basically the heart of our project (and the hardest part to figure out)

What we learned

We learned how to work with a specialized AI model using langgraph As mentioned before we learned the react framework and we further improved our understanding of the Gemini-API and FastAPI

What's next for Billify

The next step would definitely be to work on bringing this application to mobile, because right now this is a web based service. We also want to figure out a better testing process to validate our redaction process

Built With

• eel
• fastapi
• gemini-api
• javascript
• langgraph
• python
• react