As head of household, we usually pay Utility Bills without any kind of review and utility bills are not free of errors; Wrong Classification, Meter Reading Errors, Incorrect Applications of Tariffs, Late Fees, Gaps in Billing Dates, Abnormal Usage, Demand Charges, etc that make the revision very hard.

Also, people with visual impairment have faced a bigger problem, even for people without vision diseases needs help to review utility bills, that includes foreign people that do not speak the same language that utility bill.

Finally, the lack of electronic-based alternatives for bill presentment makes necessary to deal with printed documents.

What it does

The traditional way to extract data from documents is about to generate text from the document using an OCR, after that identify data entries using either image-based or text-based rules, such as regular expressions. However, this approach is a poor fit because there isn’t context or layout information. Now, thanks to Artificial Intelligence on AWS we can do it using AI-powered cloud services.

How I built it

I created an Android App with a simple UX powered by Lex. It is a simple chatbot to send Utility Bills that are processed by Lambda to pass images to Rekognition (though SNS) to extract OCR from the image. After that we can do many dirty tricks to extract the amount to pay, however we pass that data to comprehend to turns characters into syntax elements and part of speech, then we can look for the numeral (total to pay) with a confidence of 0.99+ and validate that is part of a phrase that includes nouns that meaning "total to pay".

Challenges I ran into

Create an architecture based on serverless technologies and loose coupling messaging. Also, use Amazon AI services was a challenge but, very straightforward.

Accomplishments that I'm proud of

Finishing a functional MVP that accomplish functional and Hackathon requirements.

What I learned

A lot of AWS cloud services such as: Lambda, SNS, and AI Services (lex, rekognition, translate, comprehend, polly) and aws mobile services (amplify, cognito)

What's next for ADP (Augmented Document Processing)

I'm going to create other several MVP (different kind of documents, include ML to detect errors, anomalies, frauds, etc) to continue the product development to finish a prototype, find a co-founder and register a business.

Built With

Share this project: