SplitPay

Image Processing Used
Thresholding tests

Inspiration

Every time we go out with friends, it's always a pain to figure payments for each person. Charging people through Venmo is often tedious and requires lots of time. What we wanted to do was to make the whole process either by just easily scanning a receipt and then being able to charge your friends immediately.

What it does

Our app takes a picture of a receipt and sends to a python server(that we made) which filters and manipulates the image before performing OCR. Afterwards, the OCR is parsed and the items and associated prices are sent to the main app where the user can then easily charge his friends for use of the service.

How we built it

We built the front-end of the app using meteor to allow easy reactivity and fast browsing time. Meanwhile, we optimized the graphics so that the website works great on mobile screens. Afterwards, we send the photo data to a flask server where we run combination of python, c and bash code to pre-process and then analyze the sent images. Specifically, the following operations are performed for image processing:

RGB to Binary Thresholding
Canny Edge Detection
Probabilistic Hough Lines on Canny Image
Calculation of rotation disparity to warp image
Erosion to act as a flood-fill on letters

Challenges we ran into

We ran into a lot of challenge actively getting the OCR from the receipts. Established libraries such Microsoft showed poor performance. As a result, we ended up testing and creating our own methods for preprocessing and then analyzing the images of receipts we received. We tried many different methods for different steps:

Different thresholding methods (some of which are documented below)
Different deskewing algorithms, including hough lines and bounding boxes to calculate skew angle
Different morphological operators to increase clarity/recognition of texts.

Another difficulty we ran into was implementing UI such that it would run smoothly on mobile devices.

Accomplishments that we're proud of

We're very proud of the robust parsing algorithm that we ended up creating to classify text from receipts.

What we learned

The the building of SplitPay, we learned many different techniques in machine vision. We also learned about implementing communication between two web frameworks and about the reactivity used to build Meteor.

What's next for SplitPay

In the future, we hope to continue the development of SplitPay and to make it easier to use, with easier browsing of friends and more integration with other external APIs, such as ones from Facebook, Microsoft, Uber, etc.