Evaluate LLM Cognitive Behavioral Cybersecurity Reasoning

• 

  OllaBench1 Architecture

• 

  Main Scores (the higher the better)

• 

  Wasted Resources (the lower the better)

Inspiration

The grand challenge that most CEO's care about is maintaining the right level of cybersecurity at a minimum cost as companies are not able to reduce cybersecurity risks despite their increased cybersecurity investments [1]. Fortunately, the problem can be explained via interdependent cybersecurity (IC) [2] as follows. First, optimizing cybersecurity investments in existing large interdependent systems is already a well-known non-convex difficult problem that is still yearning for new solutions. Second, smaller systems are growing in complexity and interdependence. Last, new low frequency, near simultaneous, macro-scale risks such as global pandemics, financial shocks, geopolitical conflicts have compound effects on cybersecurity.

Human factors account for half of the long-lasting challenges in IC as identified by Kianpour et al. [3], and Laszka et al. [4]. Unfortunately, human-centric research within the context of IC is under-explored while research on general IC has unrealistic assumptions about human factors. Fortunately, the dawn of Large Language Models (LLMs) promise a much efficient way to research and develop solutions to problems across domains. In cybersecurity, the Zero-trust principles require the evaluation, validation, and continuous monitoring and LLMs are no exception.

Therefore, OllaBench was born to help both researchers and application developers conveniently evaluate their LLM models within the context of cybersecurity compliance or non-compliance behaviors.

What it does

A Scenario was generated based on a peer-reviewed knowledge network of cognitive behavioral constructs for cybersecurity (the nodes) and the relationships among them (the edges). Examples of cognitive behavioral constructs include Belief, Norm, Goal, etc. There are 10,000 scenarios in the default benchmark dataset. A scenario consists of two employees (A and B). Each employee has a short description of their cognitive behavioral profile containing natural language description of cognitive behavioral construct paths (a collection of nodes and edges).

The Evaluator Model (i.e. Arctic) generates the scenarios and questions. The Evaluatee Models (i.e. any model supported by Ollama) answer the questions and are compared based on the following scores.

Which Cognitive Path question presents four options of cognitive behavioral construct paths and asks the model to select the (only) correct option that fits the cognitive behavioral profile of either employees. The Avg WCP score is the average score across a model's answers each of which receives 1 if correct and 0 otherwise.

Who is Who question asks the model to decide who is MORE/LESS compliant with information security policies based on the stated cognitive behavioral profiles. The Avg WHO score is the average score across a model's answers each of which receives 1 if correct and 0 otherwise.

Team Risk Analysis question asks the model to assess whether the risk of information security noncompliant will increase if the two employees work closely together in the same team. The Avg Team Risk score is the average score across a model's answers each of which receives 1 if correct and 0 otherwise.

Target Factor Analysis question asks the model to identify the best cognitive behavioral construct to be targeted for strengthening so that the overall information cybersecurity compliance posture of the two employees can increase. The Avg Target Factor score is the average score across a model's answers each of which receives 1 if correct and 0 otherwise.

The Average score is the average of each model's 'Avg WCP score','Avg WHO score','Avg Team Risk score','Avg Target Factor score'. The model with the highest Average score could be the best performing model. However, it may not be the case with the most efficient model which is a combination of many factors including performance metrics and wasted response metric.

Wasted Response for each response is measured by the response's tokens and the response evaluation of being incorrect. The Wasted Average score is calculated by the total wasted tokens divided by the number of wrong responses. Further resource costs in terms of time and/or money can be derived from the total wasted response value. The model with the lowest Wasted Average score can be the most efficient model (to be decided in joint consideration with other metrics).

How I built it

Potential Impacts

Given that human factors significantly contribute to enduring challenges in cybersecurity, the framework’s evaluation results will foster AI-driven innovation and efficiency in the field. For example:

• Bench-marking results will inform practitioners on which LLMs may be the best candidates for their human-centric LLM applications, reducing costs in early development phases and save costs in the long run.
• Researchers may run OllaBench after each fine-tuning of their human-centric cybersecurity models.
• Both practitioners and researchers may modify OllaBench for other projects.

The Results (so far)

What's next for OllaBench

• JUL2024 - White paper on OllaBench v.0.2 evaluation results of mainstream open-weight LLMs such as Llama3, Llama2, Orca2, Gemma, etc.
• DEC2024 - OllaBench v.0.3 release, providing datasets and benchmark scripts for evaluating LLMs’ ability to detect Cognitive Warfare manipulations in chat texts.
• JAN2025 - White paper on OllaBench v.0.3 evaluation results of mainstream open-weight LLMs
• FEB2025 - Release tutorials on how to customize OllaBench for any interdisciplinary evaluation purposes of LLMs.
• MAR2025 and forward - Running workshops on how to use OllaBench in various conferences, IEEE chapters, and university sponsored events.

Built With

• arctic
• nvidia
• ollama
• python
• streamlit