Scrabblers

Scrabblers digital poster

Scrabble Learning Model (SLM)

Morgan Lo, WaTae Mickey, Ben Kang, Moise Gasana

Inspiration Our insipration came from the scrabble tournamet and how they have to manually input the scrabble board state into the website they use here we want to automate the process using our model

Introduction Online, there exists a small but mighty community for the board game Scrabble, dedicated to following the professional Scrabble scene. Within this community, in any given year, there are likely 15-25 professional Scrabble tournaments streamed on Twitch and YouTube, each covering the top board, composed of multiple camera feeds, the main one of which is a top-down view of the Scrabble board.

These live streams include broadcasters, typically Scrabble Grandmasters, who will attempt to evaluate positions or possibilities, which often involves the use of Scrabble evaluation engines. Unfortunately, because tournament boards are not digitized in a manner like professional chess, these commentators (as well as viewers) must manually plug in values for themselves into these engines. This can be a tedious and arduous task, since a Scrabble game involves placing one hundred letter tiles.

Our project's goal is to leverage both computer vision and convolutional neural networks to convert a Scrabble board from a livestream screenshot into programmatically useful data, which could then be used for a variety of applications, such as computer analysis to return best moves, a website to display digitized live game results, or a continuous analysis bot that processes the frames from a livestream itself.

We will utilize OpenCV for basic detection of the board, locating corners, and slicing up the board to get images of each letter, and then we will apply a CNN to perform classification. As for a dataset, we intend to train the CNN with the Chars74K dataset and test our model with screenshots of scrabble boards that we manually maintain.

Methodology We utilized the Chars74K dataset to train a Convolutional Neural Network (CNN) for the task of character recognition. This dataset comprises 62 classes representing digits, uppercase, and lowercase letters, with a total of 620,992 images formatted in various styles including italic, bold, and normal fonts. To accommodate the unique requirement of recognizing blank Scrabble tiles, we extended the dataset by introducing a 63rd class for these blanks. We wrote a script, blank.py, specifically designed to generate images of blank tiles. This script created blank images by filling them with a uniform background color, matching the size and format of the images in the original dataset. These images were then saved in the Sample063 folder, effectively expanding the dataset to include this new class.

Our CNN architecture incorporated multiple convolutional layers with ReLU activation and max pooling for reducing dimensions. It began with an input layer for 128x128 pixel images, followed by convolutional layers. After max pooling, the data was flattened and processed through a dense layer of 256 neurons with 50% dropout to mitigate overfitting. The final layer, a softmax, was designed to classify the 63 character classes, including the added blank class for Scrabble tiles.

Training was facilitated using the Adam optimizer and categorical cross entropy as the loss function. We employed an augmented training approach using TensorFlow's ImageDataGenerator to enhance generalization, applying transformations such as rotations and shifts. This methodology was instrumental in developing a robust model capable of recognizing a diverse range of printed characters and blank tiles with high accuracy.

Results Quantitatively, our model achieved an accuracy range of 70%-80% over 10 epochs on the Chars74K dataset, demonstrating a reasonable capability to recognize printed characters in a controlled environment. However, these results did not translate effectively to real-world applications, as the testing accuracy on images of actual Scrabble boards was significantly lower. Notably, the model's confidence in its predictions varied considerably, and the identified characters fluctuated with changes in the amount of training. Qualitatively, while the model could output the structure of a Scrabble board, it struggled with accurate and consistent character recognition.

Challenges The primary lessons learned from this project center on the challenges of applying deep learning models to real-world tasks, especially when training data does not perfectly reflect test conditions. One major issue was the discrepancy between the training data (black and white printed letters) and the test data (colored images from Scrabble boards), which likely confused the model if it inadvertently learned to use color as a distinguishing feature. Another challenge was working with the limited power of our personal laptops. We would have to train our model for over an hour just to determine whether our testing works, which was very tedious and time consuming. Moreover, the availability and diversity of training data were major constraints. In hindsight, we think using labeled Scrabble tiles as our training data would have significantly improved our model’s performance, but data acquisition was a big challenge in itself. This lack of quality training data underscores the need for a more representative dataset.

Reflection Reflecting on the completion of the project, we are somewhat pleased with how it ultimately turned out, particularly considering the complexities involved in applying convolutional neural networks (CNNs) to real-time Scrabble board analysis. Initially, our base goal was to successfully digitize Scrabble boards from livestreams using basic computer vision techniques, reaching an accuracy of 90%. Our model achieved that accuracy on training data, but we were unable to calculate accuracy for testing data, as the testing data was unlabeled. Instead, we measured success with probability, or how confident the model was in its prediction. Unfortunately, the confidence in each image was very low.

The model generally performed as expected, but with a lot of room for improvement. Our model successfully takes a Scrabble board and splits it into images of the individual cells using computer vision techniques and classifies characters correctly, but it does not accurately classify the character on the Scrabble tile correctly. In the future, we would try to acquire labeled data of Scrabble tiles and use that data to train the model, which we believe would immensely improve our results. One pivot was the creation of a 63rd class for blank tiles, which required additional data handling and model adjustment. This was necessary to be able to classify the blank Scrabble tiles. Another pivot was when we switched from using the UC Irvine Machine Learning Repository Letter Recognition dataset to the Chars74K dataset to train our model. This was because the initial dataset was a database of character image features, not a dataset of actual images, which we needed because our testing data was images. If we were to redo the project, we would spend more time on data augmentation and possibly exploring more sophisticated CNN architectures. We would also try to find a more powerful computer to use, as training and testing on our personal laptops was a very long process.

With more time, further improvement could focus on enhancing the model's ability to generalize from the training data to real-world applications, perhaps by incorporating more diverse training examples from various Scrabble tournaments. Experimenting with different model architectures and tuning hyperparameters could potentially yield better results. Our biggest takeaways from this project include the importance of data quality. We learned a great deal about the practical challenges of deploying CNNs in real-world scenarios and the critical nature of iterative testing and adjustment in model development.

Attached is our final writeup as a PDF in the 'Additional Info' section of this Devpost, our digital poster in the image gallery of this Devpost, and the link to our GitHub repo in the 'Try it out' section of this Devpost