Nobody looked forward to Security Awareness Training (SAT). It was yet another task to do during onboarding. It was a routine yearly chore for all employees. Yes, it was important – but the employee experience wasn't great. Neither our educators nor our employees relished sitting through their end-of-year retraining.

Worse than poor employee experience, we had doubts about how effective it was. When you teach someone how to pick a good password or how to lock their laptop, they will remember what to do well the first few days. But will they still remember after a month? What about six months? And what happens when you introduce new security policies and practices? Can you be sure everyone got the memo?

What if we could replace this annual training model with something lighter, less invasive, more fun? What if instead of testing staff once a year you scatter little questions throughout the year, both as a nudge and as a way to measure how well knowledge is recalled? And what if – most importantly – you saw a measurable improvement in human behaviour when it came to security? The question was a tantalising one, and led us to create Knowledge Bot.

What it does

Knowledge sends employees a brief multi-choice question at multiple points throughout the year. This acts as both a nudge that ensures knowledge is retained in long-term memory and as a way for educators to continuously measure the workforce's awareness of a given topic.

We think this is a great way to replace or augment employee training. Questions are sent on a schedule to maximise the benefit of spaced repetition, an evidence-based memory technique that can help staff recall critical knowledge months after a training session. It also reduces the burden on educators and allows them to quantify awareness and identify areas for improvement across teams.

It's important that the experience is fun and a light touch. The questions we added use Block Kit to make questions visually interesting, using company-specific references, custom emoji and GIFs. In the Home Tab, you can see your current streak of correct questions, and can request a question to keep playing at any time.

The frequency and selection of the questions is driven by each person's proficiency with the topic. For example, a new employee who has just completed onboarding might have a head swimming with new things to remember. At the beginning, they will receive a single question every day or two. It takes a brief moment to select an answer. Each correct answer increases their proficiency level, which will increase the delay until they receive a new question. At peak proficiency, an employee will receive one question approximately every two weeks.

We've taken care to ensure Knowledge Bot is fun and non-intrusive. It respects your Slack timezone, allows you to customise which hours and days you'd like to receive questions on, and has a snooze feature which you can use if you'd like to take a break from questions.

How we built it

We used Bolt.js, TypeScript, and Postgres, and Heroku. We developed a CI/CD pipeline and added our own test harness for testing the Slack App. Code is scanned using SCA and SAST tools for security, and dependencies are monitored for known vulnerabilities. Privacy is protected by limiting information we request and store from users (the only identifying information we store is user IDs). We worked closely with beta users who gave us feedback both directly and via an anonymous survey.

Challenges we ran into

We used Knowledge Bot internally, on real security training material with a group of 40 employees. The reception from our beta testers was better than we'd hoped, and we're optimistic that Knowledge Bot solves a widespread problem in a novel way. In an anonymous survey sent to beta testers, 86% either agreed or strongly agreed it had made them more security conscious. In retrospect, it would have been better to perform a more empirical validation, measuring actual recall rather than survey sentiment.

We experimented with public leaderboards, which drove participation. We showed the top 5 each week. However, we quickly found that a few people would rise to the top and then aggressively answer questions to keep their score high, preventing others from ever hoping to secure a spot. In the end we retired the feature and stuck to a streak score an employee can see in the App's Home Tab.

On a technical level, the overall experience building the app was good. Automated testing proved somewhat difficult. We would have liked better support (libraries, documentation) for testing Bolt.js applications. Type definitions for Bolt were sometimes hard to reason about, or were slightly different from the documented/observed APIs. Bolt's tutorials weren't always up-to-date with the latest release. Socket Mode made a huge improvement to local development, and so did the new YAML configuration manifest (being able to spin up a developer-ready app via an API would have been nice). A UX problem that we ran into was being unable to directly put the power of Block Kit in the hands of question-writers. Block Kit Builder is a great tool, but not suited to a lay person. Currently, a developer and a question-writer have to work together to design the right question.

Accomplishments that we're proud of

  • Reduced the workload for the training team
  • Improved security awareness and security culture
  • Made both employees and educators happier

Anecdotally, one of the more unexpected results was witnessing employees' eyes lighting up when they received a new notification. The sentiment was, “Oh, it's time for a new question!” It was almost the same way someone might say, “Oh, today's Wordle just came out!”

What we learned

Key findings from our anonymous survey:

  • 86% said Knowledge Bot made them more security-conscious
  • 93% said it’s important to continuously verify our knowledge rather than annually
  • 93% said Knowledge Bot fits our culture and values
  • 100% said Knowledge Bot would be useful and suitable for new hires

Along the way, we learned of the wider needs for workplace training. We saw how training for compliance (e.g. HIPAA) could be improved through a model of continuous monitoring rather than annual testing. We saw a proliferation of training material across different teams and roles, showing us how we could deliver more targeted training depending on the topics most relevant to the employee.

What's next for Knowledge Bot

First off, we want to prove that Knowledge Bot improves recall, and to what extent. We've designed a small study in consultation with our education team. Two cohorts of new hires will be given Knowledge Bot along with their onboarding. Two cohorts of new hires will not. We'll compare the different groups' ability to recall the material several months later. This will start to give us an empirical basis on which to judge Knowledge Bot's effectiveness and impact.

On the feature side, we have a long write-up of possible ways to improve the experience. At the top is support for multiple topics. This would allow us to expand the material to include role-specific training (e.g. Biz Dev 101), team/project-specific training, and certification practice (e.g. Slack Certified Developer). We also want to improve the experience for educators: from designing questions to import existing learning material and analysing the data across teams and cohorts.

Built With

Share this project: