Communication is the foundation of the world, and in this modern age many people prefer to record ideas, thoughts, and information using phones and tablets. Unfortunately, non-native speakers of a language are often at a disadvantage; children and adults alike can find it quite difficult to understand and communicate, making relationships and learning an incredible challenge. We were inspired to create an app that could take any form of writing in various languages and translate it into formatted documents in other languages for other people to read and learn from. We decided to create TidyUp.
What it does
TidyUp is notetaking app that works on iOS phones and tablets. The structure of the app is simple: Any user can write directly on the notepad, in one of 13 languages, and switch to the secondary view to see what they've written (including mathematical expressions, paragraphs of text, lists, and more) appear beautifully formatted. Here they can write new text as necessary, spell and grammar check the current text, and speak into the microphone for speech-to-text functions. Using the translate function, any handwritten text can be converted into formatted text in any one of the other languages supported by the app. Using the translated text, anyone can learn to write and better understand a different language from their own. You can also share anything you've written with the share feature for both handwritten and formatted notes. Lastly, the app learns as you write to better understand how you write and translate it with more accuracy -- without the need for pre-training!
How we built it
TidyUp was created using Xamarin, WritePadSDK, and the Microsoft Azure Text Translator service. Xamarin provided the framework for creating the application for mobile devices, permitting us to write the app in C# using .Net CORE packages. The WritePadSDK was a cross-platform compatible, pre-trained recognition engine that we updated using new features, such as mathematical expression recognition and advanced multilingual support. Microsoft Azure Text Translator services allowed the app to take recognized, handwritten text and convert it to a variety of other languages. To teach the app how to understand unique handwriting styles, an automatic learning process was added as a functionality of the app. This unsupervised learning algorithm uses a neural network to classify text features and weights, automatically updating the model as the user continues to use TidyUp.
Challenges we ran into
Initially, we wanted to have the app generate PDFs of the formatted text and have the formatted text be reversibly written in your own handwriting. Numerous attempts yielded little success for the former, so we designed our own method of creating neat, formatted text that the user could save and share as needed. The latter was only partially finished, and with more time could be a profound tool in reconstructing a user's handwriting for informal messages, letters, etc. in non-native languages. We also had difficulty implementing the neural network for unsupervised learning, but with several hours of effort, we figured out how to make it possible.
Accomplishments that we're proud of
We are proud to have created an app that successfully works bidirectionally in multiple languages, supporting written notes and translations all-in-one. We are furthermore proud of augmenting the recognition engine to support several new features, most notably the ability to recognize and format mathematical expressions and formulae. We are also quite proud of our usage of a neural network to constantly retrain the model; this method of training makes it easier on the user and the programmer to create an effective and efficient app for recognition purposes.
What we learned
This was a learning experience for both members of the team; Logan had never worked with any form of OCR before and was new to cloud services, while Anirudh had never created an app before -- or for that matter ever used C#! It was a great first hackathon and taught us what to expect from future hackathons.
What's next for TidyUp
With a bit more work, TidyUp could easily be ready for release. We would want to improve the recognition engine and automatic learning system, as well as add support for more languages and separate notebook features. Publishing this app to the Apple App market (and later Google Play market) for free would make it an excellent public utility for people around the world -- both for simple notetaking and for enhancing communication as a whole!