The original file
How well the system captures data without and human input.
Even the line items!
We can read Chinese documents!
We can read even read Japanese documents. It has been trained over all major scripts and languages.
And this is independent of layout or file type.
This is the user interface.
Where it is easy to check the data extracted from a document.
Easy to add feeds to send in documents. We can use existing channels.
Documents can also be uploaded directly.
And can be edited to be scanned again from within the UI.
Challenge 5) How might we leverage technology to streamline mundane parts of the audit process that contain manual and time-consuming tasks such as vouching and reconciliations, so that auditors can save time, reduce errors and focus on higher value work?
Josh (PhD Physicist/Computer Scientist) Ashima (PhD Physicist/Computer Vision) Ben (Chartered Accountant)
The words 'mundane' and 'repetitive' seem to get mentioned constantly when talking about the audit process. Audit is just really boring! Typing in numbers is dull, time consuming, and there is so much opportunity for mistakes.
But - it's just about retrieving and matching data - so ideally it should be possible to automate. We noticed that the fact this is not automated yet mostly appears to be related to the non-uniformity of the data (which means humans are needed to read and check that different formats of the same information match). There are infinite number of possible formats for financial documents where the data distribution is extremely heterogenous. If all data was perfectly available, and nicely ordered, this job would be very much open to automation and super fast.
We don't believe we are the first people to make this observation. It seems like automated data extraction to get everything into a clean format is widely viewed as the solution here. Paper isn't going anywhere soon - as much as we might like to believe otherwise. So our question is why isn't AI extraction already used?
We found there were 3 major reasons for this: The first is the ease of use and speed of any system. The automation has to be fast and simple enough to be adopted by a wide range of users. The second is the accuracy. Most of the AI solutions achieve a maximum ccuracy of about 80-95% which is just isn't good enough when it comes to the matters of money. The third - making an AI system is extremely hard because of the complexity of the problem.
We think we have a solution that fixes all of these - and thus really removes the manual processing pain of audit using an intelligent and cognitive form of automation.
What it does
Staple is both a platform, program interface, and a machine learning system, that allows the direct extraction of data from financial and KYC document pdfs and images. The system is currently completely cloudbased, but could also be run on a private cloud.
Easy extraction of data
Documents may be fed into the system directly or via whatsapp, dropbox, email, wechat or an API. One can also upload thousands of documents together for automated data extraction. All the data extracted from documents is automatically transferred to the associated account of the user on staple's platform.
Fast and accurate results
On the Staple platform, a user may then view the document, and extracted data in an online user interface where they can edit and verify the extraction. Edits are fed back to the system to retrain our AI model. This ensures 100% accuracy. For extracting data from 1 document, it takes about 10-30 seconds. Thanks to parallel cloud services, for extracting thousands of documents, it also takes about 10-30 seconds. This all happens regardless of language, or the format of the doucment. No templates for documents are required.
Automated integration to account systems
Once all the data extracted from in a document is verified - the data can be immediately exported directly into an account systems such as Xero or Quickbooks, or into a full ERP system like SAP. Alternatively, a user can also download the extracted data into the format of their choice.
If the user does not wish to validate the data, and simply trusts the AI when it reports a set level of confidence - that is also possible. Thus less human involvement is needed over time. This system happens with thousands, to millions of documents at once. Meaning the data extraction and standardization of the audit process is very very fast.
Being cloudbased systems
As everything is currently available on the cloud - this means it is possible to connect to the system instantaneously. It also means we can make inprovements to the product continuously without affecting user experience.
Specific application to audit activities
Amazingly - everything is (more or less) live!
You can try a demo here of the invoice and receipt scanning tech! This allows you so see how well the system can extract from PDFs and images in seconds!
And the user interface can be seen here. This shows how users can make edits to extracted data.
The extraction is also available as a direct API - which means you can even merge Staple's technology into your own platform. The documentation on how to use the API is here.
We want to stress - all of this (except the UI) is live to try now and was built by Josh and Ashima.
How we built it
We built the initial data extraction tech on the backend using our own work around openCV2 framework, an openly available OCR, the SpaCy package for natural language processing, and a custom trained ML model.
This was then wrapped in flask, docker, NginX and elastic-beanstalk to make it available as an API via amazon webservices code. Documentation was made using Postman.
The platform user interface is still not completed but we are continuing to build using ReactJS, Apollo, GraphQL, and Redux.
Challenges we ran into
There are two major issues:
1) Getting data into the system quickly
The data input needs to be fast and easy. We thus added integrations to whatsapp/wechat/dropbox to allow easy feeding - as well as the option to manually upload data. These can be screenshots/pdfs/photos.
2) The accuracy of the data
Getting the AI close to 100% turned out to be really, really hard. Ours is pretty good now - we estimate in the mid 90%s range. But we realized that it doesn't matter how fast or cheap the extraction is if there are any mistakes at all in the output - people won't forgive the system. Even if we make 1 mistake in 1000 documents we found that people wouldn't want to use the system.
So adding the User Interface (UI) is required to let people sign off on documents, and that system also needs to be really fast and intuitive to use. However, the data extraction system already needs to be pretty good before they will use the system at all (or else it's not saving time when you first start using it). It's a catch-22 and we could only get past it by putting a lot of effort into training to scanning tech ourselves. We fel we have acheived this - The system works for any format of invoice, and even Chinese language documents now work well (although handwritten data is still a little hit or miss).
Accomplishments that we're proud of
We compared our solution to others online (episoft, sphyt, Veryfi, receiptbank, expensify). We are actually significantly better in many ways: 1) We can pull data from photos. 2) It doesn't use humans to do initial data extraction. 3) It reads from any major language and script. 4) It gets the line items. 5) It's infinitely scalable. 6) It can do any kind of document! It would be easy for us to scale to add parking tickets/certificates/etc. 7) We don't require any kind of template.
We are also very proud of the User interface - we spent a lot of time looking at what was currently in the market and thinking about why it wasn't being used. After this we tried to implement the cleanest and simplist system we could in the design.
What we learned
Some errors are going to happen in an AI model - especially one reading complex data from photos. Getting the accuracy close to 100% (including a human accountant) is difficult, but people will not be forgiving of an AI compared to a person. Thus having a human in the loop is necessary. Having someone in the loop validating results turns out to be a benefit as it helps train our model, and thus improves the tech while continuing to create a huge time save. But we have to be already pretty good to make people use the system initially.
We also found that allowing multiple methods of feeding into the system allows all parts of any business to submit data in realtime. Sending in data by whatapp make it easy for anyone to use the system.
We also learned a lot about deployment and servers! The tech stack we chose is very modern; GraphQL is only a couple of years old and not widely employed in products yet - but it has made the communication very fast. Elastic-beanstalk means we can make everything infinitely scalable.
The affect on Audit
This means no more repetitive tasks. No more typing in numbers or descriptions. This makes data extraction faster and thus allows accountants to spend time on higher value activities - which involve real decision making based on correctly extracted and formatted data.
This also means all data input can be completely realtime - a client takes a photo of a document and it goes straight into the account system. This means that when completing filings, there is no spike in workload. We don't need more apps to acheive this - we can use existing channels to make onboading clients to the system very simple.
Finally when performing audit sampling - it is easy to use the system for checking any number of documents. Simply pick some number of documents and take a photo (or get the pdf/scan), run them through the system, and the data is extracted and ready to cross reference against the ERP/account system in one step.
This feature can also aid in the completion of multiple steps of the audit process, as outlined below:
- Vouching of invoices against POs to ensure expenditure is authorised. Staple extracts data from bills and invoices and converts it into clean, structured data for automation purposes. But it can also cross check invoices data against appropriate documentary authorisation such as system-based purchase orders. This can reduce the incidence of unauthorised expenditure for clients, and create a value added service that auditors can offer to clients.
- Vouching of delivery notes, service reports, completion certificates and other documents to substantiate revenue. Typically auditors must test revenue by vouching goods or services rendered against documentary evidence. This process involves the review of unstructured information (documents) and comparing it with structured information (in the system). Staple can bridge this divide and extract large volumes of documentary evidence, convert it into csv or some other preferred format, to later compare against system records.
- Expedite searches for unrecorded liabilities. Auditors must verify the completeness of liabilities recorded on any given client's books. This typically involves the identification and collection of bills and invoices in an office location, then checking that these expenses/liabilities are recorded in the client's books. An auditor could use Staple to ingest large volumes of documents identified, convert it into structured data and automatically compare potentially unrecorded liabilities against the liabilities recorded in the system.
- Identification of potential fraud, part of an auditor's statutory duty. Beyond its powerful data extraction capabilities, Staple has a layer of intelligence that can recognise anomalies in data. For example, with the extraction of large datasets, our technology can be trained to recognise irregular transactions directly from the source, for example: 1) where invoices are received with conveniently round values, like $10,000 precisely. 2) where duplicate invoices are received from the same or different vendors for the same service, product, or for the same PO. 3) where false vendors are issuing invoices. 4) where other anomalies arise in a client's spending activity.
Staple can flag any and all of the above!
What's next for Staple
Staple platform is going to revolutionize the accounting industry with AI. Although attempts have been made at solving the problem already by previous companies, none of them fill the small gaps that connects the whole pipeline and provide an end-to-end solution for accounting workflows. With the completion of User interface on Staple's platform, we can extract millions of documents under a minute, ready for a human to review and ensure 100% accuracy in an extremely intuitive environment. This would also ensure the adoption of this AI solution to book-keepers, outsourced accounts and big MNC's worldwide. Hopefully over time, this feedback from human will retrain our AI models, thereby eliminating the need for human intervention at all. Auditors can thus save their valuable time, effort and energy into more challenging and demanding tasks rather than spending hours on manual data entry.
As our models develop, we plan on offering more extensive AI products to service the accounting industry. With these tools, accounting firms, audit firms and corporate service providers could diversify their service offering beyond standard bookkeeping, audit and compliance offerings.
1) Advisory on spend policies. Staple can identify unusual line items within documents, e.g. if a "pina colada" appears on an invoice supposedly for office supplies. Our technology can red flag expenses that are outside of policy, and auditors could benefit from this tech as an add-on service to on-sell to clients. 2) Advisory on spend management. Staple can spot inefficiencies in spending patterns and recommend where clients are over-spending on certain line items, and/or recommend better value vendors of certain goods or services.