LeggoDutch

Logo
Index Page
1
2
Receipt used
3
4
5

Inspiration

Everyone has probably experienced the pain of splitting the bill after a great dinner with your friends at least once in your life. No one really wants to pick up the bill because it is always so hard to go after people to remind them to pay for their share of the food. On top of that, no one is really in the mood to do mathematical calculations and send Venmo requests after a full meal. LeggoDutch recognizes this problem and aims to assist users to allocate bills to the right person with our receipt recognition model. By splitting the receipt up into different line items, the user can assign them to different people. Once the food is assigned, LeggoDutch will calculate the meal costs and send reminder texts to the different parties to return the amount of money to the user.

What it does & How we built it

After the user submits a photo of the receipt, LeggoDutch's model will preprocess the image to de-skew and soften the lighting for better text processing. It then uses text detection and recognition to pick up fragments from the image before converting them to texts. The text detection module uses ClovaAI's CRAFT PyTorch engine in conjunction with Tesseract 4.0 to parse the image into text characters. Afterward, another module comes into play to analyze the text and break it down into food items and their corresponding prices before displaying them to the user.

The front-end interface will handle the user input and organize the list of friends and mobile numbers. Following this, it will allow the user to assign food items to the correct person by clicking on the right name on the dropdown menus. LeggoDutch uses the Twilio API to disseminate text reminders to the other people. Because we are currently using the trial version of Twilio API, we are only able to send text messages to registered numbers.

Challenges we ran into

ClovaAI's CRAFT PyTorch was not enough to give us accurate predictions on the prices of the different items. It sometimes gets confused by the period sign between the dollar and cents figures. Hence, we had to put in more data sets to further train the model. The quality of the text recognition is also heavily dependent on the orientation of the image as well as the contrast in lighting between the receipt and its background. We struggled to capture the four corners of the receipt under such circumstances and can only transform the image to a limited extent. However, with the combined help from our different machine learning components (image processing, image segmentation, and text recognition), we are able to cover some of the flaws of our image processing technique. Because the Twilio API's trial version does not allow us to send text messages to numbers that are not registered on our accounts, we are unable to implement our product widely until we secure funding for a proper Twilio subscription.

Accomplishments that we're proud of

We are happy to be able to improve on an existing trained model from ClovaAI by developing our own training process. Even though we were not able to implement a fail-proof image cropping and transformation technique, we managed to cover the flaws of the cropping technique with the other components' capabilities. We are also happy to have been able to deploy a machine learning model along with a Flask API online. It's our first time to have completed an end-to-end program.

What we learned

We learned how to design our data set to better train our model. At the same time, we learned other machine learning techniques for image transformations and their relative advantages and disadvantages over each other. It is also the first time that we have attempted to develop and deploy a machine learning model. We learned a lot about containerization and how to work with environment variables.

What's next for LeggoDutch

We plan to implement Venmo API for direct request and payment, as well as a user account system to connect friends and store their information. This would help to facilitate future transactions and improve the overall user experience. At the same time, we hope to train our Pytorch model to respond to receipts from different countries so that LeggoDutch can be used all over the world.