ChemType

What it does

ChemType is a note taking tool for chemists which turns a handwritten organic chemical structure into a typeset image. The application uses machine learning optical structure recognition technology to identify up to 9 different structures from a user generated handwritten image. Within the app, the user hand draws a structure which is then sent to the machine learning algorithm to be processed, and is returned as a typeset image.

How I built it

ChemType uses a machine learning algorithm called a neural network classifier for optical structure recognition. To generate training examples, we hand drew 40 diagrams of each structure with pen and paper. Because we had such few training examples, great care was taken to ensure high training data quality by preprocessing the images. Dimensionality reduction, reducing each image from millions of dimensions to only 96 dimensions, allowed the neural network to focus on the broad structural features of the molecule. Additionally, we applied a background correction to denoise the images. We then developed and tested the neural network using Microsoft Azure Machine Learning and used Azure's web service to transfer raw data and results to and from our iPad app. The main job of the app was to get the user generated image and send it to Azure, then collect the result of the neural network algorithm from Azure and give the user the corresponding image.

What's next for ChemType

With more training data ChemType can eventually be able to recognize hundreds of structures. Adding further machine learning algorithms will allow the app to recognize structures it has never been taught before by recognizing their chemical functional groups, as well as recognizing text. Because all the heavy-duty computation is loaded on the cloud, the app has a great potential to be scaled up without putting a burden on iOS or the user's local machine. With the right algorithm, ChemType could make chemistry note taking entirely digital, completely bypassing the need for conventional molecule editing software.

Built With

• azure
• python
• swift