-
-
woops
-
Inspiration
"We're all installing AI tools as fast as they ship — agents, MCPs, whatever hit the front page this week — with zero idea if they're safe to run. LLM / AI Safety Check gates them: six data-backed signals per tool, worst-signal-wins, red/yellow/green. And here's the proof it works — we ran it on a dataset frozen at June 2023, using only what was knowable then, and it flagged mlflow before it was mass-exploited and AutoGPT with a completely clean CVE sheet. The data was screaming; this makes sure somebody's listening."
What it does
You let people find popular trending agents, MCPs, and LLM tools, after it gathers popular tools and their descriptions, it tells you whether each one is safe to run on your machine, as a 🔴 red / 🟡 yellow / 🟢 green verdict with evidence attached.
Concretely, for each tool it:
- Finds it in the supply-chain dataset (deps.dev/OSV) by asking questions in plain English
- Figures out what it actually is — NLP dedupes name variants, drops false positives, and tags dangerous capabilities (does it execute generated code? expose a server?)
- Grades it on six signals — known CVEs, dangerous capability, staleness, blast radius, upstream health, identity trust — combined worst-signal-wins
- Checks the verdict against reality — a web lookup shows what actually happened to the tool after the data snapshot Shows its work — every claim traces back to the SQL query or source URL that produced it ## How I built it The pipeline is a fixed LangGraph graph rather than an autonomous agent — a deliberate choice for determinism and auditability: discover → classify → gate → hindsight → synthesize → report. Data access is 100% CRAFT. Every question the pipeline asks — top AI tools by stars, advisory counts in the trailing year, reverse dependents — goes through CRAFT's MCP tools (generate_sql → execute_query) as natural language. I never hand-wrote an analytics query. This had a side effect that became a feature: since CRAFT returns the SQL it generates, every verdict ships with a complete audit trail (sql_queries.txt) for free. Classification is Nemotron on Nebius Token Factory. Raw dataset results are messy — autogpt vs auto-gpt, tutorials matching AI keywords, masscan matching "scan." Nemotron dedupes entities, filters false positives, and — most importantly — tags dangerous capabilities by reading descriptions and docs. This is the load-bearing NLP: there's no executes_untrusted_code column in deps.dev, so the signal that red-flags AutoGPT can only come from unstructured text. ## Challenges I ran into FABLE DOWNGRADED ME TO OPUS BECAUSE I WAS SEARCHING FOR CVES SQUASHING MY PARALLEL AGENTS ## Accomplishments that I'm proud of eh, i got it done ## What I learned Have another premium model available to run tests ## What's next for Ryan Kill Fable tokens. Have less than 24 hours.
Built With
- craft