We're tired of sorting through physical post we don't care about. It's much easier if you just sort it away and store the stuff you care about in the cloud.
What it does
Any letters, leaflets etc posted into the mailbox are photographed and analysed using the Azure CustomVision API, and any relevant text extracted using OCR techniques. The resulting classification is used to sort mail into two bins, and inform the choices made in emailing (eg. confidence changes the language used in the email, and the largest text is included in the subject).
How we built it
It's powered by a Qualcomm DragonBoard 410C attached to a Logitech webcam which captures images of the post. The resulting classification decision is sent to an Arduino board, which controls motors in order to sort it into the appropriate bin. Rather than constantly polling, the Arduino notifies the DragonBoard when new post is entered through the slot.
Challenges we ran into
.NET Core is difficult to set up on Linux distributions which it does not explicitly support -- we relied on the prebuilt binaries supplied by Microsoft in order to run our project.
Additionally, it does not support the full range of .NET features, which proved problematic when using Serial ports to communicate with the Arduino. This was also an issue for webcam capture, but we delegated this task to a command-line utility which could be used as input to the main C# code.
Accomplishments that we're proud of
Each of us had very different experiences in the project, as we solved different problems.
Alan: The Azure integration and sample training data. It was nice to learn a new API, and gain new skills.
Will: Solving the concurrency bug which prevented reading from the serial port while simultaneously testing with Azure. The resulting solution used parallelism and locks in order to prevent duplicate readings, and ensure that the user was notified the proper number of times.
Jakub: Making the hardware work. The kit was not at all designed for what we used it for, but it all worked out.
What we learned
Proper use of .NET Core, and how it works on different platforms. Also, Azure and how it makes complex and efficient machine learning platforms easy to produce.
What's next for SmartMailbox
This could be an interesting prototype for a real IoT device; much of the technology would need to be changed in order to deal with bursts of mail, different sized post, and the need for faster response times in a real production environment. However, much of the classification technology can be reused due to the modular structure employed in the project; and it can be easily extended in order to expose additional information to new APIs