LINDEF: Lightweight Real-Time ML Intrusion Defense

LINDEF is a lightweight ML system that detects network attacks in real time, classifies attack types, and supports faster and more resource-efficient containment using low-latency intrusion defense.

Comment

Inspiration

I was inspired to build LINDEF after learning how many cybersecurity tools are designed for large organizations with expensive infrastructure, dedicated security teams, and constant maintenance. Smaller organizations, schools, and local networks still face real cyber threats, but they may not have access to the same level of protection.

I wanted to explore whether machine learning could make intrusion detection more lightweight and accessible. Instead of only focusing on accuracy, I wanted LINDEF to also consider practical factors like false positive rate, latency, RAM usage, and response actions. That led me to build LINDEF as a lightweight network intrusion detection and containment framework.

What it does

LINDEF detects suspicious network traffic and classifies the likely attack type.

It uses a two-stage machine learning pipeline:

  1. Binary Detection Model
    The first model classifies each network flow as either benign or malicious.

  2. Attack Classification Model
    If traffic is flagged as malicious, the second model predicts the likely attack type, such as neptune, smurf, nmap, portsweep, guess_passwd, httptunnel, and other attacks.

After classification, LINDEF maps the predicted attack type to a severity level and a recommended response action, such as:

  • ALLOW
  • BLOCK_IP
  • THROTTLE_IP
  • ISOLATE_HOST

I also built a Streamlit simulation dashboard that displays recent detections, attack type, severity, confidence, recommended response, and summary charts. The current dashboard demo uses sample flow data generated from the training/testing pipeline, but it is structured to support a future live workflow using TShark and CICFlowMeter.

How I built it

I built LINDEF using Python, Google Colab, scikit-learn, pandas, NumPy, imbalanced-learn, Joblib, Matplotlib, Seaborn, psutil, Streamlit, CICFlowMeter, and TShark/Wireshark.

The model was trained using three public intrusion detection datasets:

  • NSL-KDD
  • UNSW-NB15
  • CIC-IDS

I combined these datasets to increase attack coverage, improve generalization, and expose the model to a wider range of network traffic patterns. After loading the data, I cleaned missing and infinite values, removed duplicate rows, dropped leakage-prone columns such as IP addresses and timestamps, encoded categorical features, scaled the final feature set, and used SMOTE to reduce class imbalance.

LINDEF trains two Random Forest models:

  • A binary classifier for normal vs. attack detection
  • A multi-class classifier for attack type classification

I evaluated the system using accuracy, precision, recall, F1-score, false positive rate, ROC-AUC, latency, CPU usage, and RAM usage. The binary model reached 99.83% accuracy, while the attack classification model reached 94.15% accuracy. I also compared LINDEF against signature-based IDS, anomaly-based IDS, cloud endpoint detection, and rule-based IDS methods using benchmark ranges summarized in my project poster.

Challenges I ran into

One of the biggest challenges was combining three intrusion detection datasets that were not originally designed to work together. NSL-KDD, UNSW-NB15, and CIC-IDS all use different feature sets, formats, labels, and column structures. Because of this, I had to carefully clean the data, standardize columns, remove leakage-prone fields, and align the final feature space so the models could train correctly.

Another major challenge was live feature extraction. CICFlowMeter can extract flow-based features from packet captures, but it does not directly recreate every NSL-KDD or UNSW-NB15 feature from raw network traffic. This made it difficult to perfectly match live traffic features to the combined training feature set. To handle this, I saved the final training feature list and designed the dashboard to align incoming flow data to the expected feature columns.

Getting CICFlowMeter working consistently was also difficult. Different versions and setups behave differently, and connecting packet capture, flow extraction, feature alignment, scaling, and model prediction into one smooth pipeline required a lot of debugging.

I also ran into broader limitations. LINDEF has not yet been tested in large enterprise or high-bandwidth networks, and testing was done on limited hardware. Like many supervised machine learning systems, it may struggle with zero-day attacks or traffic intentionally crafted to evade detection. The model is also not fully transparent, so future work should add explainability tools to better understand why specific flows are classified as attacks. These limitations line up with the limitations I identified in my project poster.

Accomplishments that I'm proud of

One of the biggest accomplishments of LINDEF is that I built a complete intrusion detection pipeline instead of only training a model. The project includes dataset preprocessing, feature cleaning, binary detection, attack classification, benchmark comparison, model artifact saving, and a dashboard demo.

I am especially proud that LINDEF performed strongly while staying lightweight. The binary classification model reached 99.83% accuracy with a 0.21% false positive rate, while the attack classification model reached 94.15% accuracy with a 0.37% false positive rate. The models also maintained relatively low RAM usage compared to several alternative IDS approaches.

Another accomplishment was combining multiple intrusion detection datasets into one training pipeline. Since NSL-KDD, UNSW-NB15, and CIC-IDS have different feature structures and labels, getting them to work together required careful cleaning, alignment, and preprocessing.

I am also proud of the Streamlit dashboard demo because it makes the project easier to understand visually. Instead of only showing model metrics, the dashboard shows detections, attack types, severity levels, confidence, and recommended response actions such as ALLOW, BLOCK_IP, THROTTLE_IP, and ISOLATE_HOST.

Overall, I am proud that LINDEF connects machine learning, cybersecurity, performance benchmarking, and a practical user interface into one project.

What I learned

Through LINDEF, I learned that building a cybersecurity machine learning system is much more than just training a model. A working intrusion detection pipeline requires dataset cleaning, feature engineering, class balancing, model evaluation, feature alignment, dashboard design, and deployment planning.

I also learned that accuracy alone is not enough. For an intrusion detection system to be useful, it also needs a low false positive rate, reasonable latency, manageable RAM usage, and reliable feature extraction. That is why I evaluated LINDEF using not only accuracy, precision, recall, and F1-score, but also false positive rate, ROC-AUC, average latency, CPU usage, and RAM usage.

The biggest technical lesson was how difficult it is to combine cybersecurity datasets. Each dataset had different features and labels, so I had to think carefully about preprocessing and feature alignment. I also learned that live detection is harder than simulation because packet captures must be converted into the same kind of flow features the model expects.

Most importantly, I learned how important it is to be honest about limitations. LINDEF performed strongly in testing, but it still needs more live testing, larger-network testing, and feature extraction improvements before it could be considered ready for real deployment.

What's next for LINDEF

The next step for LINDEF is improving the live detection pipeline. The current dashboard demo uses sample flow data, but future versions will focus on making the live workflow more reliable using TShark for packet capture and CICFlowMeter or another flow extraction tool for feature generation.

I also want to test LINDEF in larger and higher-bandwidth environments. My current testing was limited to one machine, so future testing should evaluate how the system performs on different hardware and in more realistic network conditions. This would help measure whether LINDEF can maintain strong accuracy, low false positive rates, and manageable latency under heavier traffic.

Another future improvement is expanding the training data. Adding newer network traffic datasets and more enterprise-style traffic could help LINDEF generalize better to real-world environments and reduce weaknesses against unfamiliar attacks.

I also plan to improve the model by testing ensemble methods, tuning detection thresholds, and adding explainability tools. Explainability would make the system more transparent by showing which features influenced a detection decision.

Finally, LINDEF’s containment logic can be expanded. Instead of only assigning simple responses like BLOCK_IP or ISOLATE_HOST, future versions could consider attack confidence, repeated behavior, source patterns, and network context before recommending a response.

Built With

Share this project:

Updates