Vigilant

What it does

Vigilant is a software vulnerability intelligence system that helps security teams identify which software vulnerabilities deserve attention and take action faster.

Traditional security tools often prioritize vulnerabilities by severity, but a severe vulnerability isn't necessarily one that attackers will exploit. Vigilant uses machine learning to predict exploitation risk, identify overlooked threats, and connect those predictions to engineering workflows.

On our 2025 test set, Vigilant identified 10 subsequently exploited vulnerabilities in its top 25 predictions, compared with just 2 for EPSS alone.

How we transformed the data

We started by recreating the rules of the 2026 competition scoreboard using historical data.

The scoreboard only includes vulnerabilities that were not already on CISA's Known Exploited Vulnerabilities (KEV) list at their designated snapshot date. This matters because predicting an attack that's already known isn't useful.

Applying the same approach to 2024–2025 gave us 82,430 vulnerabilities, of which only 122 were subsequently added to KEV, a positive rate of just 0.148%.

This made the problem especially challenging: we needed to identify a tiny number of meaningful threats among tens of thousands of candidates.

Preventing data leakage

One of our biggest challenges was ensuring our models couldn't accidentally see the future.

Our primary features were historical EPSS scores and exploit indicators from ExploitDB, Metasploit, Nuclei, and GitHub, taken from each vulnerability's designated monthly snapshot. Because February 2025 had approximately 14,500 missing records, we used March snapshots for those vulnerabilities as an explicit exception.

We also investigated the NVD data, which comes from a November 2026 snapshot. Some fields, including product information, references, and updated severity scores, may have been added after exploitation.

In an early experiment, including these fields produced a promising 0.32 PR-AUC in shuffled cross-validation, but performance dropped to 0.11 on a chronological 2024-to-2025 test.

This showed how easily future information can make a model look better than it really is.

We therefore separated features by when they are generally available: historical exploit signals, vulnerability metadata such as descriptions and CVSS, and fields likely to reflect later analysis. We excluded post-analysis fields and treated potentially updated metadata cautiously.

Choosing our models

Rather than relying on a single algorithm, we experimented with six models that approach the problem differently.

Two logistic regression models analyzed vulnerability descriptions. Random Forest and LightGBM learned patterns from structured vulnerability features. A second LightGBM used 30-to-1 random undersampling to handle the extremely small number of positive examples.

We also experimented with hard-negative sampling, but found that it could teach the model misleading relationships, including treating high EPSS scores as a signal of safety.

Finally, we used LambdaMART, a learning-to-rank model, to compare vulnerabilities within their publication months.

This was particularly important because EPSS changed its underlying model in March 2025, making raw scores across different periods difficult to compare directly.

Each of our six models generates a score. We convert these into monthly rankings and combine them using learned weights to produce our final ensemble.

Evaluating our model

We evaluated Vigilant on vulnerabilities from 2025, using 2024 data for the earlier training period.

Of the 55 vulnerabilities subsequently added to KEV, our ensemble placed 10 in its top 25 predictions.

That's 40% precision in the top 25, compared with just 8% for EPSS alone, which identified only 2.

Our ensemble also achieved a PR-AUC of 0.165, compared with 0.031 for EPSS. Shuffled cross-validation averaged 0.154 PR-AUC, but we emphasize chronological evaluation because it better represents how the model would perform on future vulnerabilities.

These results suggest that combining different predictive signals can identify threats that conventional prioritization methods overlook.

From Prediction to Remediation

Predicting vulnerabilities is only half the problem. Security teams still need to determine whether a threat affects their systems and coordinate a fix.

Vigilant is designed to connect those steps into a single workflow.

Our dashboard surfaces high-priority vulnerabilities, explains why they were flagged, and matches them against an organization's software inventory. Rather than flooding engineers with notifications, an escalation engine filters findings based on prediction rank, affected assets, and business criticality.

Actionable findings can be sent to Jira for tracking, while Slack alerts notify the appropriate team when attention is needed.

For vulnerabilities that can be addressed through code, a coding agent can investigate the affected repository and propose a draft pull request. Entire captures the agent's development history, making its changes and decisions easier for engineers to inspect.

The engineer remains responsible for reviewing and approving any proposed fix.

Our goal is to move beyond simply telling teams what might be vulnerable, toward helping them understand what matters and what to do next.

What's next

Vulnerability patterns change constantly. EPSS updates its models, new software becomes popular, and attackers shift their attention between products and vendors.

A model that performs well today can quietly become less effective over time.

To address this, we plan to retrain Vigilant as new exploitation labels become available and evaluate it continuously on later time periods.

We also want to expand the remediation workflow with better asset matching, smarter alert thresholds, and more reliable automated patch suggestions.

Throughout these improvements, we'll preserve the principle behind our modeling approach: use information available at prediction time, avoid post-exploitation leakage, and evaluate on genuinely future data.

Ultimately, we want Vigilant to become a system that doesn't just identify emerging threats, but helps engineering teams respond to them before they become incidents.