Challenge 07: Prediction of ticket processing times

Inspiration

Since 2018, I've been working in computing centres with ticket systems. I also like to try out NLP and NN approaches. This challenge provided my with the opportunity to develop code and to combine both.

What it does

First, the code reads the given .xlsx file, extracts 200 rows of validation data. Second, it preprocesses the ticket title texts, calculates a Latent Dirichlet Allocation (LDA) topic model, extracts features and plots diagrams for visual feature inspection. Finally, the code tries to find a good ReLU neural network model, calculates the RMSE for 1000 rows of test data and the 200 rows of validation data.

How I built it

I developed the code using PyCharm (Community Edition) for development and testing of functions as well as Jupyter Notebooks. I used Python 3.7.4 and among others the Python libraries pandas (0.25.3), numpy (1.18.1), matplotlib (3.1.2), nltk (3.5), scikit-learn (0.23.1), gensim (3.8.3) and torch (1.4.0).

Challenges I ran into

Many of the tickets contain different types of numbers like version numbers, telephone numbers, years, dates or other numbers and I needed to replace them. The ticket titles also contain some synonyms like 'passwort' and 'kennwort' and I needed to replace the most frequent synonyms. The ticket processing times in minutes range from 0 to 140000 and because 0 values are actually occurring quite frequently as processing time in minutes, calculating and predicting the log value was not applicable, so I calculated the 4rth root. The torch library by default works fine with float values but not with the double values which I had as target values after calculating the 4rth root, so I needed to round the values to not run into conversion errors.

Accomplishments I'm proud of

I preprocessed the title texts in a way, I find the word stems in the topics fit quite well together.

What I learned

(1) Ticket titles may not say much about the processing time needed. In some tickets the problem or requirement may not be well described. (2) There may be a range of different causes which result in the issue described by the ticket creator, and processing times for the different causes which need to be fixed may vary. In other words, the cause of the problem may be unknown to the creator of the ticket, so the ticket title may not be that much related to the time needed for processing the ticket.

What's next

Quantizing the processing times and applying NLP to the ticket texts instead of the ticket titles may lead to better predictions. Also adding vectors from Google BERT to the input by using the Python libraries bert-serving-server and bert-serving-client may further improve the results.

Built With

• jupyter-notebook
• python