Inspiration

We were deeply inspired by the "Quick, Draw!" game developed by Google in 2016. This game pushed the boundaries of machine learning, where a neural network was used to identify doodles drawn by users from all around the world. The idea of a computer being able to recognize human-drawn sketches in a game-like setup captivated us. Moreover, realizing the advances in machine learning from 2016 to 2023, we were eager to take on this challenge and see how much more accurate and quick we could get in predicting doodles.

What it does.

"Pictionary Plunge" is a machine learning model tailored to interpret and predict what a user is drawing in real-time based on the strokes made. Just like a Pictionary game, as a user sketches, our model analyzes the sequence of strokes and tries to predict the drawing category as swiftly as possible. By leveraging the multidimensional array representation of the strokes, our model makes real-time inferences and provides immediate feedback.

How we built it

Leveraging the power of TensorFlow and its high-level Keras API, we made use of a model called "SketchRNN". Here's a breakdown:

Data Handling and Loading: We built our solution to be easily initialized with a provided dataset path. Using the os module, we've made sure the data exists and assigned unique indices to each drawing category available in the dataset. This index-to-name mapping is pivotal for our prediction.

Constructing the Encoder: The encoder serves as the initial part of our model. It accepts a sequence of strokes and aims to compress this information into a fixed-size context vector. The encoder uses LSTM (Long Short-Term Memory) layers, a kind of recurrent neural network (RNN), which is ideal for sequence data like strokes in drawing. Post LSTM layers, two dense layers predict the mean (mu) and log-variance (logvar) which represent the compressed information of the input strokes.

Constructing the Decoder: The decoder's responsibility is to take the context vector from the encoder and produce a sequence of output. To achieve this, we used another LSTM layer, which is conditioned on the compressed information. It generates a sequence based on the context and the sequence it has generated so far.

SketchRNN Model Composition: Our primary model, SketchRNN, combines the encoder and decoder. It first encodes the input strokes into a context vector (also performing a reparameterization trick which is crucial for Variational Autoencoders). Following that, the decoder is invoked to generate the output sequence. Finally, a dense layer maps the decoder's output to the prediction of what the drawing represents.

Drawing Interpretation and Prediction: Within the Solution class, the guess method takes the x and y values of strokes, processes them into a format suitable for our SketchRNN, and produces a prediction. This prediction identifies the drawing's category based on the trained neural network.

Model Initialization: We've also provided a blueprint for how one might initialize the entire solution using pre-trained model weights.

Challenges we ran into

Sequence Data Processing: Handling sequential data like strokes can be challenging due to its varying length and nature.

Balancing Encoder and Decoder: Ensuring that the encoder produces a meaningful context vector and that the decoder can use this vector effectively was tricky.

Reparameterization Trick: Implementing and understanding the need for the reparameterization trick (used in VAEs) to sample from the latent space was a complex challenge.

Accomplishments that we're proud of

End-to-End Model: Building a fully functional end-to-end solution that takes stroke data and produces a meaningful prediction.

Effective Use of LSTM: Leveraging LSTMs to handle the sequential nature of drawing data.

Simplicity and Efficiency: Despite the complexity behind the scenes, our solution provides a simple interface for initializing, interpreting new drawings, and producing predictions.

What we learned

Complexities of Sequence Data: We delved deep into the challenges posed by sequence data and learned the nuances of handling such data effectively using neural networks.

Variational Autoencoders (VAEs): We got hands-on experience with VAEs, understanding their structure, and their use-case in problems like ours.

Effective Model Integration: Integrating various components of our model, like the encoder and decoder, into a seamless whole was a valuable learning experience.