ThreatLens — Autonomous GitHub Vulnerability Intelligence
Inspiration
Every security team faces the same problem: there are thousands of new vulnerabilities published every day, and no one has time to read them all. The average organization has hundreds of repositories, each with dozens of dependencies, and the question "are we actually affected by this CVE?" takes a human analyst hours to answer manually.
We wanted to build something that answers that question automatically — not just collecting vulnerability data, but actively hunting through real codebases, understanding what they depend on, and surfacing only what actually matters. A security operations center that runs itself.
The secondary inspiration was purely technical: we wanted to see if ClickHouse's columnar storage could make real-time threat analytics feel instant at a scale where traditional databases would struggle. Spoiler: it does.
What It Does
ThreatLens is an autonomous threat intelligence agent that:
Scans any GitHub user or org — fetches all public repos and reads every dependency file (
package.json,requirements.txt,go.mod,Cargo.toml,pom.xml, and more)Cross-references against live vuln databases — queries the OSV API and NVD for every dependency found, identifying which exact versions are vulnerable
Triages with AI — every vulnerability gets scored by LLaMA 3.3 70B via Groq: severity 1–10 with reasoning, threat classification, plain- English summary, and a decision (IGNORE / MONITOR / ALERT)
Stores everything in ClickHouse — columnar storage purpose-built for the kind of time-series analytical queries that security teams actually run: "which repos got worse this week?", "what's our critical count trend?", "how fast is our mean triage time?"
Displays it on a live SOC-style dashboard — split panel with a scrollable CVE feed on the left, repo intelligence with per-repo vuln breakdowns on the right, and analytics tabs showing ClickHouse query benchmarks in real time
Fires alarms autonomously — when a CRITICAL vulnerability is found in a scanned repo, the browser generates an alarm tone via Web Audio API, a red banner drops from the top, the tab title changes to show the alert count, and the affected repo pulses red — all without any human intervention
How We Built It
Backend — FastAPI + APScheduler
The agent runs on a 5-minute scheduler via APScheduler. Each run:
- Hits NVD, GitHub Advisory, Shodan InternetDB, and HaveIBeenPwned APIs
- Normalises raw data into a unified threat schema
- Sends each threat to Groq for AI triage
- Bulk-inserts everything into ClickHouse Cloud
For repo scanning, we use the GitHub public API (no auth required for public
repos) to fetch repo lists, then pull dependency files directly from
raw.githubusercontent.com. The OSV API (api.osv.dev/v1/query) does the
package-to-CVE matching — it's free, requires no API key, and covers npm,
PyPI, Go, Rust, Maven, and more.
Storage — ClickHouse Cloud
Three tables drive the entire product:
-- Every vulnerability ever found, queryable in milliseconds
CREATE TABLE threat_events (
id UUID,
timestamp DateTime64(3),
source LowCardinality(String),
severity LowCardinality(String),
cvss_score Float32,
package_name String,
vuln_id String,
ai_summary String,
langfuse_trace_id String
) ENGINE = MergeTree()
ORDER BY (timestamp, severity, source);
-- Per-repo vulnerability records
CREATE TABLE repo_vulns (
scan_id UUID,
username String,
repo_name String,
package_name String,
installed_version String,
vuln_id String,
severity LowCardinality(String),
cvss_score Float32,
fix_version String
) ENGINE = MergeTree()
ORDER BY (username, repo_name, severity);
-- Agent run history
CREATE TABLE agent_runs (
id UUID,
timestamp DateTime64(3),
duration_ms UInt32,
threats_found UInt32,
langfuse_trace_id String
) ENGINE = MergeTree()
ORDER BY timestamp;
The ORDER BY on (timestamp, severity, source) means ClickHouse
pre-sorts data on disk in exactly the order security analytics queries need.
A query like:
SELECT severity, count() as cnt, avg(cvss_score) as avg_cvss
FROM threat_events
WHERE timestamp > now() - INTERVAL 48 HOUR
GROUP BY severity
ORDER BY cnt DESC
runs in < 40ms over tens of thousands of rows because ClickHouse only reads the three columns touched by the query, skipping all others entirely. That's the columnar advantage — and we show it live on the dashboard's benchmark panel.
Observability — Langfuse
Every single LLM call is wrapped in a Langfuse trace. This means for any vulnerability on the dashboard, you can click through to see:
- The exact prompt sent to the model
- The raw model output
- Token counts and latency
- The severity score and triage decision as structured metadata
In security tooling, auditability isn't optional. You cannot have a black box making risk decisions. Langfuse makes every AI decision in ThreatLens fully inspectable and replayable.
with langfuse.trace(name="threat-triage", tags=["source:nvd"]) as trace:
generation = trace.generation(
name="severity-score",
model="llama-3.3-70b-versatile",
input=prompt,
)
response = groq_client.chat.completions.create(...)
generation.end(output=response.choices[0].message.content)
trace.update(metadata={"severity": result["severity"],
"decision": result["decision"]})
Frontend — React + Vite + Recharts
Dark SOC terminal theme throughout: #0a0e1a background, JetBrains Mono
for all data display, neon green (#00ff88) accents, severity-coded colors
everywhere. The layout is a 3-zone app shell:
- Left panel (300px fixed): scrollable CVE feed, each card color-coded by severity with compact CVSS badge, package version, and one-line description
- Main area (tabbed): Repos / Analytics / Agent Log / CVE Intel
- Alarm system: Web Audio API generates a programmatic alarm tone (no audio files), critical banners, animated repo cards, and live tab title updates
Challenges We Faced
Rate limits without API keys — most of our data sources have rate limits that tighten without authentication. We built a graceful degradation layer where the agent falls back to realistic seeded data when live APIs are throttled, so the dashboard is never empty during a demo.
ClickHouse schema design for query speed — getting the ORDER BY keys
right matters enormously. Our first schema ordered by id (UUID), which
made time-range queries slow because the data wasn't physically sorted by
time. Switching to ORDER BY (timestamp, severity) dropped our benchmark
query from ~300ms to ~35ms on the same dataset.
Dependency file parsing across ecosystems — every package manager has a
different format. package.json has dependencies and devDependencies.
requirements.txt has version specifiers like requests>=2.28.0. go.mod
uses a completely different syntax. Building a normaliser that handles all of
these reliably — and fails gracefully on malformed files — took more time
than expected.
Making autonomy real, not theatrical — it's easy to build a dashboard that looks like a security tool. It's harder to build one where the agent genuinely does something without human input. The key design decision was: the agent doesn't just collect and display — it makes a triage decision (IGNORE / MONITOR / ALERT) on every threat, and that decision drives what gets surfaced, alarmed, and prioritised. The human never has to touch it for the system to be useful.
What We Learned
ClickHouse's columnar model is genuinely different — not just "faster Postgres." The physical data layout on disk changes which queries are fast at a fundamental level. $O(n)$ row scans become $O(k)$ column scans where $k \ll n$.
Langfuse makes AI systems trustworthy — observability on LLM calls is as important as observability on API calls. Without it, you have no idea why the model scored something a 9 vs a 7.
The OSV API is underrated — free, no key, covers every major package ecosystem, and returns structured vulnerability data that maps cleanly to dependency versions. Better for package-level vuln matching than NVD alone.
Autonomous ≠ automatic — a system that runs on a schedule and displays data is automatic. A system that makes decisions, prioritises, and changes its own output based on what it finds is autonomous. The difference is whether the AI output has consequences.
Built With
python fastapi apscheduler clickhouse-cloud langfuse groq
llama-3.3-70b react vite recharts tailwindcss osv-api
nvd-api github-api docker web-audio-api
Log in or sign up for Devpost to join the conversation.