ML Puzzle Solver

Why solve puzzles when your computer can solve them for you?

Jigsaw puzzles are always fun to solve. But you know what's even better? Having a machine learning algorithm to solve it for you! Our machine learning model does just that, leaving you with beautifully solved puzzles.

What it does

Our model will take in 128x128 images that are scrambled into 2x2 puzzles. Our model will predict the correct locations, or indices, of the pieces, and it will then output a string that indicates the correct indices of the puzzle pieces based on the input. These indices can be used to view the original, unscrambled image.

How we built it

After extracting each puzzle piece from the original image, we pass each of them through the model, which is a neural network. The model will extract the features and learn the useful image features for us. In this case, edge and corner features are the most useful for finding the correct puzzle configuration. Since these features emerge within the first few layers of a neural network, we chose a shallow neural network for efficiency and simplicity.

We wanted to train the model with symmetrical weighting on each image tile, so we used time distributed layers, which perform the same transform to each array in a mult-dimensional array. We also padded our input image to preserve edge information.

After that, we flattened the final layer to merge the feature vectors. Then, we passed this vector through a feed-forward network, then we reshape it into a matrix of 4x4 (we need 4 vectors to predict the 4 scores for each position, which is a 4x4 matrix). Since this is a multi-class classification problem, we used sparse-categorial-crossentropy for our loss function. The optimizer that we utilized for our neural nets is Adam. When building our neural nets, we used the relu activation function, and for the last layer, we used a softmax to choose a definitive prediction.

Challenges and Roadblocks

When loading and training the model, we had memory limitations. To overcome this, we converted our .ipynb file to a .py file, and ran it on a machine with more RAM.

Accomplishments that we're proud of

As students that are getting started in Deep Learning/Machine Learning, we worked together to create a neural network that effectively solved the jigsaw classification problem. We gained exposure to neural networks, understood the underlining theories behind it, as well as utilizing libraries and frameworks in python like tensorflow and keras to create a model that effectively solved the puzzles.

What we learned

We learned that normalization and pre-processing data before training our model is a very important process. Without the right format of data, it can affect the model's performance greatly. We also learned that loading data and fitting the model is a memory-intensive process, so optimization should be a key design consideration when utilizing Deep Learning/Machine Learning problems.

What's next for ML Puzzle Solver

For future directions, we would love to expand our model to something that can handle larger puzzle pieces (for example, 3x3, 4x4,...). If we have more time, we would also train our model on a larger training set, so we can achieve better accuracy and reduce all the noise that might affect our final performance.