High Signal

Inspiration

Over 92,000 vulnerabilities have been published in 2026 alone — more than double all of 2024 — yet fewer than 1 in 250 is ever confirmed to be used in a real attack. Security teams can't patch everything, and the usual answer, "sort by severity," tells you how bad an attack could be, not whether anyone will actually attempt it. IBM's "Before the Breach" challenge framed exactly the question their clients keep asking: what do we fix first? We wanted to connect that prediction to something an engineer can act on, not just a ranked list.

What it does

High Signal is a local workbench for prediction-guided vulnerability investigation. It does two linked things:

Forecasts exploitation. It ranks all 72,670 benchmark CVEs by their likelihood of a future addition to CISA's Known Exploited Vulnerabilities (KEV) list, using only information available at each CVE's historical snapshot cutoff, and exports a top500.csv.
Turns those priorities into investigations. You point it at a public IBM repository at a pinned commit. It inventories dependencies, matches them to advisories via OSV, joins the forecast scores, and ranks what to look at first. For a qualified fixture it then runs a bounded, isolated check that produces real evidence — and reports the forecast rank and the test outcome separately, so a prediction is never mistaken for a proof.

A candidate's outcome is always one of matched, reproduced, not_reproduced, or inconclusive. A failed or incomplete check stays inconclusive — it never silently becomes a "clean" result.

How we built it

A FastAPI + SSE backend with a DuckDB + Parquet data layer, served to a single-user local terminal UI (plain HTML/CSS/JS) with a live file tree and an ASCII evidence-network graph. The work split into three lanes:

Forecasting — a dependency-free categorical model over historical EPSS percentile bins and public-exploit flags, with strict temporal train/dev/holdout splits to avoid leakage. We reproduced all 72,670 forecasts byte-for-byte across two machines.
Repository intake & ranking — pinned-commit manifest parsing (npm lockfiles, Python requirements), OSV advisory matching, and a deterministic queue. On top of it, SiftRank uses IBM Granite 4 (3B), running locally through Ollama with a bounded Go adapter, to comparatively re-rank candidates with full request/response trace evidence.
Evidence — a qualified vulnerable/patched fixture (IBM/mcp-context-forge, CVE-2026-53710) that runs one reviewed assertion inside a locked-down Docker container: no network, read-only root, non-root user, dropped capabilities, and CPU/memory/process limits. The identical check passes on the patched commit and trips on the vulnerable one, and every run saves its command, image digest, timings, and raw output for hash-verified replay.

IBM Granite also generates the evidence-linked explanation for each finding, citing only saved evidence — model prose can never change a recorded test outcome.

Challenges we ran into

Leakage was the hardest part. Today's NVD data is backfilled, so fields like reference counts or review status can quietly leak the answer for 2026 CVEs. Keeping the historical forecast strictly to each CVE's snapshot cutoff — and keeping present-day data as display-only context — took real discipline.
Honest evaluation. We have no 2026 outcome labels, so we can't claim a measured hit rate. We held ourselves to reporting only what the held-out historical splits actually showed, including where our model did not beat the EPSS baseline.
Running known-vulnerable code safely. Building a check that proves a difference between two revisions, while guaranteeing it can't touch the host or the network, meant pinning image digests and verifying the container's isolation before importing any source.
Three people, one contract. We split into forecasting, harness, and integration lanes with a versioned JSON/Python contract between them, so each lane could move independently and hand off validated artifacts.

Accomplishments that we're proud of

A complete, reproducible forecast over all 72,670 benchmark CVEs with an honest, leakage-audited evaluation.
A real, isolated vulnerable-vs-patched check on an actual IBM advisory, with replayable raw evidence — not a mock.
Local-only use of IBM Granite for both ranking and explanation, with bounded budgets and full trace capture.
A disciplined separation of prediction, applicability, and tested outcome throughout the UI and the data model, so nothing overstates what was actually established.
Over 1,000 integration tests across the forecast/candidate/harness boundaries.

What we learned

Severity is not the same as risk, and a package-version match is not the same as reachable, exploitable code. The most useful thing a tool like this can do is be honest about that gap: show the forecast, show the evidence, and keep them clearly labeled. We also learned how much careful plumbing "keep it fair" actually requires when the ground-truth data has hindsight baked in.

What's next for High Signal

Measure real prioritization uplift: compare prediction-guided vs severity-first investigation under an identical scan budget, on time-to-first-confirmed-finding.
Broaden beyond one fixture to more IBM repositories and more harness adapters.
Add source-level reachability analysis so applicability is backed by call-path evidence, not just version matches.
Calibrate the forecast into probabilities once post-event outcome labels become available.