Siemens Graph the Building Challenge

Results of first iteration of neural network
Results of first iteration of neural network
Results of first iteration of neural network
OpenCv KeyPoint Matching seemed promising but we ran into time constraints
Successfull Template Matching example
False template recognition 1 (Tries to place Temperature sensor)
False template recognition 2 (Tries to place Temperature sensor)
Unthinned hand Sketch
Post processed thinned sketch and contors plotted on top of sketch
Shape detection (Was not able to predict shapes reliably enough when considering hand sketches)

Inspiration

The inspiration for this project came from the Siemens graph the building challenge Pitch. To us, it seemed like an exciting idea that could be both useful in helping the environment as well as an interesting new technical skill we could hopefully pick up.

What it does

Our hack takes a hydraulic schematic and analysis all given components within the schematic while mapping their connection to allow for conversion into a more easily readable node diagram.

How I built it

We first tried building programme by loading an input image into python, removing noise and turning it into a black and white picture using OpenCV. It was very important to us that we would be able to properly interpret hand sketches as well as computer images which caused us to spend a lot of time at the pre-processing stage to ensure that hand sketches were properly and clearly processed.

We then mapped areas of interest on the image which would represent the location of hydraulic components. After this, we boundary boxed the components and removed them from the image which should have left us with just the connecting pipes which would then have been mapped to their corresponding components allowing for post-processing analysis and transformation of the input image into the more desired node diagram.

However, due to challenges described in the section below, we knew that we would not be able to develop a solution using this approach in the time left so we decided to reduce the scope of our solution to just identifying all components within a hydraulic diagram over deep learning approach and building more functionality and analysis on top of this if we could.

We deployed a Yolo neural network with a medium-sized data set of both hand-drawn and computer-generated sketches of the hydraulic symbols created by us overnight. This model could classify the given symbol in a given sketch to a reasonable degree and we are confident that this would have been improved with more time for a bigger data set.

Challenges I ran into

As we had both never worked with machine learning before, we knew that we were in for a challenge. We initially struggled a bit with figuring out how to use OpenCV and how to deploy its various features best to solve the challenge.

We initially followed a Template matching approach to map hydraulic symbols to input images but ran into issues with thresholding to confirm whether a component was present at all or multiple times. We tried to solve this problem by using different norming methods but as we could not manage to accurately and reliably confirm the presence of a component in a diagram we decided to pivot our approach to the challenge to a deep learning approach due to time constraints.

As mentioned earlier, machine learning is a new topic to us, so we had to learn the basics of machine learning and neural networks and play around with some classification models until we felt confident to deploy a model towards the challenge of the problem.

Accomplishments that I'm proud of

First time ever having experience with image processing, Machine learning and programming in python. This was also the first time we deployed a deep learning model.

What I learned

-The fundamentals and mathematics of Machine Learning. -How to use and train neural networks. -A basic understanding of python. -Image processing.

What's next for Siemens Graph the Building Challenge

Use vastly bigger and more diversified datasets for the hydraulic symbols. The first iteration of our classification seems quite promising so we think that with further training we could get it to be more accurate.