RIA: RCBC Inquiry Assistant

RIA assists clients in answering bank-related inquiries by classifying their intent and redirecting them to proper channels. Using LLM, this app classifies English and Filipino questions.

Comment

Inspiration

In the wake of the global pandemic, the field of customer service experience has significantly changed. Support teams have seen an increase in the queries they have received. Automation using AI has emerged as a pivotal tool in meeting these demands. However, the challenge lies in extending these advancements to languages with fewer resources, such as Filipino. There are very few existing datasets and pretrained models in Filipino, hindering the ability of deep learning models to generalize well.

Thus, we introduce the RCBC Inquiry Assistant or RIA, an intent classifier for banking queries trained in both English and Filipino languages. The vision is to streamline the customer service experience to be inclusive and efficient with the use of large language models.

What it does

RIA assists clients in answering bank-related inquiries by classifying their intent and redirecting them to proper channels. Using a large language model, this app is able to classify both English and Filipino questions.

Ideally, the model should be embedded within the bank’s website. We envision the user to visit the bank’s website, type in their query in a search bar or chatbox, and the model will route the query to the right team or send the links that can help address the question.

How we built it

In order to build our model, we turn to transfer learning techniques, which enable knowledge learned from one task or language to be transferred to another. We used Universal Language Model Finetuning or ULMFiT as our transfer learning technique, which consists of language model pretraining and finetuning, and classifier finetuning. For finetuning, we used the Banking77 dataset, an intent detection dataset for banking customer inquiries with 10,003 entries for training and validation. We trained and served the model using Databricks and deployed it as an app in streamlit.

Challenges we ran into

  1. Finding domain and language specific datasets is challenging. We needed to use the Google Translate API to translate the Banking77 dataset into Filipino prior training.
  2. During the experimentation phase, some multimodal LLMs do not generalize well on domain specific datasets unless we do further finetuning and feed it with domain specific datasets.
  3. Creating language models for low-resource languages like Filipino is challenging. Fine-tuning the language model and classifier allowed us to capture the context of Filipino texts more accurately.

Accomplishments that we're proud of

  1. We were able to do end-to-end model deployment and maximized Databricks' features.
  2. We were able to integrate Databricks with streamlit and github seamlessly.
  3. RIA is the first LLM-based application in our company. And to the best of our knowledge, RIA is also the first LLM-based application to classify banking intent/questions in Filipino language.

What we learned

  1. For Filipino datasets, it is important to pay special attention to banking jargon for better model performance.
  2. There is a learning curve for the team in using Databricks since we just learned it throughout the course of the hackathon. Despite this gap, the team was able to deliver a working LLM application within the given timeframe.
  3. Serving a model in Databricks is straightforward using the user interface and deploying it via streamlit.io is seamless. The Databricks support team helped a lot in answering the team’s questions.

What's next for RIA: RCBC Inquiry Assistant

For the bank's processes:

  1. Short Term: Improve the functionality of the company's website search button to deliver relevant results, especially for queries in the Filipino language
  2. Medium Term: Implement an automated system for categorizing the intent of complaints or inquiries received through social media, streamlining the response process.
  3. Long Term: Open doors to other LLM-based applications that can help improve internal processes e.g. knowledge-based embedded systems

For bank employees:

  1. Short Term: Enable customer service agents with the tools to efficiently access specific knowledge base, enabling them to promptly address client inquiries.
  2. Medium Term: Influence team organization structure within customer service based on categories derived from the study to promote more efficient processes
  3. Long Term: Introduce LLM-based applications to other groups within the organization to scale the benefits obtained

Built With

Share this project:

Updates