PANCAKESS — Devpost description

Short description

Security teams face thousands of new vulnerabilities, but only a small fraction are ever exploited, and ranking by severity score alone misses most of them. PANCAKESS uses a time-safe machine learning model and IBM Granite text embeddings to predict which CVEs attackers will target. In our 2025 backtest it ranked later-exploited CVEs better than CVSS or EPSS (AUC 0.874 vs 0.707–0.768), and every CVE comes with the reasons behind its score and AI-generated plain-English summaries. We also audited the data for leakage, so results reflect only what a real team would have known at the time.

Inspiration

Every month brings tens of thousands of new CVEs, and no team can patch them all. The usual shortcut, CVSS severity, measures how bad a flaw could be, not whether anyone will actually exploit it. We wanted a fix-first list that answers the question security teams really have: which of these will attackers use, before they use it?

What it does

PANCAKESS ranks all 72,670 CVEs on the 2026 scoreboard by how likely each is to be exploited (CISA KEV is our proxy for exploitation), using only information known by the end of each CVE's snapshot month. The dashboard at pancakess.work shows a searchable ranked queue with P0–P2 priorities. Each CVE has a drawer with the signals pushing its score up or down and an AI-generated IBM Granite summary, clearly labelled. An optional Gemini executive brief never changes the ranking.

How we built it

We built a time-safe pipeline over NVD, CISA KEV and EPSS data. Features are exploit-code signals, each CVE issuer's past KEV history, and masked description text. A gradient-boosted model (scikit-learn) combines them with an IBM Granite embedding text score, and the top 500 is capped at 50 rows per issuer to stay robust to 2026 drift. Results are served by a FastAPI backend to a React, Vite, Tailwind and Three.js dashboard, hosted on Nginx with Let's Encrypt. Every published number is registered in a claims ledger and re-checked by an independent auditor. Code: github.com/Saad5A/Datathon2k26.

Challenges we ran into

  • Leakage. Fields in the NVD record quietly give away the answer: all 52 later-exploited CVEs in our 2025 test were marked "Analyzed" and had product lists, against 50.1% of the others for "Analyzed". We excluded these fields, and we treated every too-good result as leakage until proven otherwise.
  • Tiny test set. With only 52 exploited CVEs, one hit is about 2%. We report intervals, and we found that our AUC gain over the best baseline is borderline, and the top 500 roughly ties it.

Accomplishments that we're proud of

  • Higher AUC than CVSS, EPSS and exploit-code-then-EPSS in the 2025 backtest (0.874 vs 0.707–0.768), with the median exploited CVE ranked around 1,288 against 2,499–7,960 for the baselines.
  • A leakage audit and a shuffled-label control (AUC near chance, 0.532 and 0.506).
  • A claims ledger: every number we publish was re-derived by an independent auditor, and we make no 2026 accuracy claim, because no 2026 labels exist.

What we learned

Honest evaluation is harder than modelling: the biggest early gains were leakage, not insight. A pre-registered rule decided that IBM Granite would be adopted on a tie, and we say plainly that it gave no measurable gain over the model without it. Small numbers of positives demand uncertainty bars, not headlines.

What's next for PANCAKESS

Monthly re-scoring: re-run the time-safe pipeline as new KEV and EPSS data arrive, then measure real 2026 hits once labels exist. Solution integration: feed the existing prediction API and CSV export into ticketing and SIEM tools (Jira, ServiceNow, Splunk) as a daily fix-first queue.

Built with

Python, pandas, scikit-learn, IBM Granite (open Hugging Face weights, cross-checked on watsonx.ai), Google Gemini (optional), FastAPI, React, Vite, Tailwind CSS, Three.js, Recharts, TanStack, Nginx, Let's Encrypt.

Built With

Share this project:

Updates

Submission history