A few years back, I worked as a customer service representative. Businesses often receive a high volume of customer service tickets every day, and it can be difficult to manually sort and categorize each ticket in a timely and accurate manner. This can lead to delays in response times, frustrated customers, and decreased satisfaction with the overall customer service experience.

TicketSensei can help improve the efficiency and accuracy of customer service operations. By automating the process of categorizing tickets, customer service agents can focus on providing personalized and effective solutions to customers. This can lead to improved response times, reduced customer frustration, and increased customer satisfaction and loyalty.

In addition, the data generated by the system can be used to identify trends and patterns in customer inquiries, allowing businesses to proactively address common issues and improve their products and services. This can lead to improved customer satisfaction and increased loyalty over time.

What it does

The customer service ticket classification model will be used to predict the category of a new ticket based on its content. When a new ticket arrives, the model will analyze the text and assign it a category based on the patterns it has learned from the training data. This category can then be used to assign the ticket to the appropriate team or individual for further processing. By automating the ticket classification process, the model can help to streamline customer service operations and increase customer satisfaction.

How I built it

The first step was data collection whereby we collected and preprocess the ticket data, labeling the tickets with their respective categories and storing it in CSV format. Next, a Cohere account was signed up for and an API key was created to access Cohere's playground. The development environment was also set up by installing the Cohere Python library and loading the preprocessed data into a pandas DataFrame. After the data was loaded, a Text Classifier model was initialized using the Cohere Python library. The number of categories to classify tickets into and any other parameters, such as the learning rate and batch size, were also specified.

After initializing the model, it was trained on labeled data using the fit() method of the Text Classifier model. The number of epochs and batch size for training were specified. The model learned to classify the tickets into their respective categories based on the labeled data.

To evaluate the model's performance, an evaluation() method for the Text Classifier model was used to evaluate its performance on a validation set, the model was fine-tuned by providing additional training data.

The model was tested using a set of unseen data to evaluate its accuracy and performance and the confidence levels refer to the model's estimated probability that a particular label accurately describes the input text. The higher the confidence level, the more certain the model is about its prediction.

Looking at the list of labels and their corresponding confidence levels, we can see that the model is most confident about the fourth and eighth examples, both of which predict the label "Product issue" with confidence levels of 0.95969296 and 0.99550474, respectively. In contrast, the model is least confident about the first example, which predicts the label "Account management" with a confidence level of only 0.460344.

It's worth noting that even when the model is not very confident about a particular label, it still provides a probability estimate for that label. For example, in the first example, although the model is not very confident that "Account management" is the correct label (with a confidence level of only 0.460344), it still provides estimates for the other labels as well, with "Billing Inquiry", "Product issue", and "Technical support" all receiving probability estimates of varying confidence levels.

import cohere
from cohere.responses.classify import Example
co = cohere.Client('n0JwfDrhBmEla44Q2MkQv6DiXsuJUoD5TLMTpcvb') # This is your trial API key
response = co.classify(
  inputs=["How can I set up a 3rd party contribution to my Account?", "Where can I find my unused Coins?", "How do I link my return with my partners?", "Do I need to complete a return if I moved to Canada this year?", "Can I set up a business account on your platform?", "Do you offer self-directed and managed investments?", "What is the current interest rate in your savings account?", "I am unable to log in to my account.", "My account was charged twice for the same purchase.", "I need help updating my billing information."],
  examples=[Example("My order was cancelled but I was not refunded yet.", "Technical support"), Example("I am having trouble resetting my password", "Technical support"), Example(" Can you tell me more about your return policy?", "Technical support"), Example("Interest rates are going up - does this impact the interest rate in my savings account?", "Technical support"), Example("What is the best option for a student savings account?", "Technical support"), Example("My family situation is changing and I need to update my risk profile for my equity investments", "Billing inquiry"), Example("Where can I see the YTD return in my investment account?", "Billing inquiry"), Example("How can I change my beneficiaries of my investment accounts?", "Billing inquiry"), Example("Is crypto an option for my investment account?", "Billing inquiry"), Example("How often do you rebalance your investment portfolios?", "Billing inquiry"), Example("What is the monthly fee on the investment accounts?", "Billing inquiry"), Example("How can I withdraw funds from my investment account?", "Billing inquiry"), Example("Can I buy stocks and ETFs listed on non Canadian exchanges?", "Billing inquiry"), Example("How can I minimize my tax exposure?", "Product issue"), Example("I\"m going to be late filing my ${currentYear - 1} tax returns. Is there a penalty?", "Product issue"), Example("I\'m going to have a baby in November - what happens to my taxes?", "Product issue"), Example("How can I see my ${currentYear - 2} tax assessment?", "Product issue"), Example("When will I get my tax refund back?", "Product issue"), Example("How much does it cost to use your tax filing platform?", "Product issue"), Example("I\'d like to increase my monthly RRSP contributions to my RRSP", "Account management"), Example("I want to take advantage of the First Time Home Buyers program and take money out of my RRSP.  How does the program work?", "Account management"), Example("What is the ${currentYear} RRSP limit?", "Account management"), Example("Does your system ensure I won\'t overcontribute to my RRSP?", "Account management"), Example("How do I set up employer contributions to my RRSP", "Account management"), Example("My subscription was cancelled but I still have access to the service.", "Account management")])
print('The confidence levels of the labels are: {}'.format(response.classifications))

Challenges I ran into

As junior machine learners, we faced several challenges when building NLP models on Cohere Playground. The main challenges we encountered included understanding fundamental NLP concepts, such as tokenization and named entity recognition, data preprocessing to ensure our data was in a suitable format, selecting the right model from Cohere Playground's pre-trained models, fine-tuning hyperparameters to ensure optimal performance, and interpreting results effectively. These challenges required us to collaborate, conduct extensive research, and leverage Cohere Playground's tools to overcome them and build effective NLP models.

Accomplishments that I'm proud of

Designing and developing a working NLP model using Cohere Playground is a significant accomplishment for us as a team. The process required effective collaboration, extensive research, and leveraging Cohere Playground's tools to overcome various challenges and build an effective NLP model.

What I learned

Through the process, the team gained a deeper understanding of fundamental NLP concepts, and we are willing to improve our data preprocessing and hyperparameter tuning skills, and enhanced collaboration and communication skills, all of which will be valuable in future NLP and team-based projects.

What's next for Customer Service Ticket Classification

This could be applying techniques such as transfer learning to improve the model's generalizability and further refine the model's hyperparameters and training data to improve its performance. This could involve experimenting with different hyperparameter settings, incorporating additional training data, or refining the existing data.

Share this project: