As an international NGO with democracy and governance programs in more than 60 countries with funding from the US Government and other agencies, NDI is the target of a great number of attacks from Advanced Persistent Threats (APT). We set out to use Splunk to build an application that would alert our engineering team to DDoS attacks in order to protect NDI and its partners around the world and ensure continuous access to critical information and services.
The National Democratic Institute has created a Splunk application that detects potential denial-of-service (DoS) attacks against a web server. This application has the ability to send out email alerts to system administrators when the event is triggered from a potential DoS attack.
How it works
NDI has created an application using Splunk to enable DoS detection. At a high level, the application monitors all traffic to the web server and uses triggers to enable events (such as sending out email alerts) when the application detects that the server may be under a DoS attack. For testing, we used a basic Drupal website running under the Apache web service. Apache logs all access requests to the server which Splunk monitors in real-time.
At its simplest level, when a specific IP address makes more than X requests over a span of Y seconds (these variables can be tweaked based on the type of server and the expected normal level of traffic to the server), the DOSAttackDetection application will trigger and cause predetermined events to fire. For our test setup, we used e-mail alerts to send emails to system administrators alerting them that a possible DoS attempt is occurring. The e-mail contains information about the originating IP addresses along with the number of requests made by those IPs over predetermined time windows.
The app also includes a dashboard made of of the visualizations described below.
Data Visualization Components
The DDos Attack Detection application developed by NDI has many useful data visualizations to help system administrators quickly assess performance and security metrics. When creating data visualizations, it’s important not to overload the end-user with too many charts and graphs. The data that is presented should be concise and quickly actionable by system administrators. In the event of a DoS attack, it’s critical that system administrators have the information needed to mitigate an attack and respond quickly with a cogent plan to thwart the attack and create networking rules that will prevent future attacks from the same source.
The first data visualization is a pie-chart that shows the top IP addresses that are “noisy.” These are ip addresses that have made the most requests to the server over a specific time window. While these ip addresses may not be the source of an attack, if an attack is in progress, this chart will show which IP addresses are participating in the DoS attack.
Status Codes Visualization
The “HTTP Status Codes” bar chart shows a breakdown of the status codes returned from the web server. Status codes are three digits codes that are used to classify the type of response from a web server. Generally, status codes starting with 2 (2xx) mean that the request completed successfully. Status codes starting with 3 (3xx) are used when redirection of a request is involved. 4xx status codes are errors that are caused by the client-side (the computer requesting a resource from the web server). These codes range from the well known “404 - Page not found” code to other statuses that may be thrown because the client is not authorized (403) or is making too many requests (429). 5xx status codes are reserved for server-side errors. If the server is unable to complete a response successfully due to a problem on the server itself, a 5xx status code is returned.
Top IP Requesters
The top IP requesters chart shows which IP addresses have made the most requests over a 23 hour period (this period can be adjusted). This chart is useful to see exact counts and percentage of resources used based on source IP addresses. An IP address could be a legitimate heavy consumer of resources. This helps system administrators plan for future server loads and scaling up infrastructure to meet the growing needs of a successful web site.
Challenges we faced
Learning how to use the splunk filters took some time since there are so many different commands and options available for alerts, etc. The Splunk documentation also has some areas that are a bit unclear such as integrating with the Splunk API.
What's next for the DDoS Attack Detection App
We will continue to expand our application to handle distributed DoS attacks and to enhance our error monitoring (status code errors, etc.)
The NDI Splunk DoS detection app is an exciting addition to NDI’s core suite of security tools that enable NDI to respond swiftly to DoS attacks by giving system administrators real-time alerts that are quickly actionable by our internal security team. By leveraging this data, we can strengthen our internal networking security protocols and policies to assist in mitigating any future attacks.
Splunk allows NDI to be proactive when dealing with server security and allows us to continue providing the highest level of system security to our employees, partners and end-users of our products.