Crucible
The Problem Nobody Is Auditing
A skill can be perfectly safe and still teach an agent to work badly.
No malware. No exposed secrets. No prompt injection. It passes every security scanner you own.
And it says:
"Retries MUST continue until success."
Four lines of Markdown. An agent installs it, and from then on, it retries forever against an unavailable API — with no bound and no way to verify the obligation it just declared.
Nobody attacked anything. Someone wrote a bad methodology, and an agent believed it.
Crucible audits precisely this category of failure: not whether a skill is malicious, but whether the methodology it installs into an agent is sound, verifiable, and internally consistent.
Security scanners inspect threats. Crucible inspects methods.
Why Now?
People are installing hundreds of third-party skills into their AI agents.
One collection we analyzed contains 1,057 skills. Another contains 818.
Nobody can reasonably review all of them line by line.
Traditional security scanners catch conspicuous failures: malware, exposed credentials, prompt injection.
But what about a skill that passes every security check and still teaches an agent to retry indefinitely, declare obligations it never verifies, or promise capabilities its instructions cannot deliver?
Those failures don't necessarily trigger alarms.
They remain installed, quietly influencing agent behavior.
The methodologies we give our agents have become infrastructure. Crucible brings infrastructure-grade auditing to those methodologies.
What Crucible Does
Crucible compiles an agent skill's declared methodology into source-addressable records and audits those records deterministically.
It doesn't simply ask a language model whether a skill looks reasonable.
It creates inspectable findings, ties them to source evidence, and produces reproducible decision records.
It also tests its own detector by deliberately introducing defects into fixtures and verifying whether those defects are detected.
The model proposes. The deterministic gate decides.
A language model can suggest a merge, repair, or independent assessment. It cannot override the deterministic decision process.
We tested that separation directly.
The same merge, executed twice at temperature zero, produced two differently worded skills from the model.
The deterministic gate returned the same verdict, the same reason, and the same finding in both executions.
Change the model's wording. The decision remains unchanged.
Powered by NVIDIA Nemotron and Nebius Token Factory
Crucible uses NVIDIA Nemotron 3 Super 120B through Nebius Token Factory as its live proposal engine.
Not a mocked integration. Not a demonstration substitute.
The model participated in real experiments, including:
- Merging redundant skills while preserving deterministic acceptance criteria.
- Proposing repairs that transformed failing properties into passing ones.
- Confirming redundancy candidates across real skill collections.
- Demonstrating that variation in generated language does not change deterministic gate decisions.
This integration is fundamental to the architecture.
Crucible deliberately places a capable language model in the proposer role while reserving final authority for independently verifiable rules.
Nebius Token Factory enabled repeated live experiments against external corpora, demonstrating this separation in practice.
Nemotron provides generative capability. Crucible provides deterministic accountability.
Why We Built It This Way
Crucible began with a methodological question, not a conventional security-scanning approach.
Its intellectual foundation comes from the social sciences and the Peircean tradition of inquiry: identifying what a claim commits us to, examining the evidence supporting it, and testing whether that evidence survives attempts at refutation.
Crucible applies that discipline to the instructions autonomous agents execute.
A SKILL.md file is not merely documentation. It can define operational behavior, obligations, verification procedures, and failure-handling strategies.
Those declarations deserve scrutiny.
The central question is not simply:
"Is this instruction dangerous?"
It is:
"Does this methodology actually support what it claims to accomplish?"
That distinction also explains Crucible's architecture.
Language models are powerful generators and interpreters, but persuasive explanations are not equivalent to verified conclusions.
Crucible separates proposal generation from decision authority so that confidence in an audit does not depend on confidence in a model's narrative.
Proven on Real-World Data
337 Duplicate Skills. Every Candidate Verified.
On a third-party corpus containing 692 skills, Crucible identified 337 redundancy candidates.
We manually verified every candidate.
337 confirmed true positives.
The collection published skills twice: once under their original names and once with category-prefixed names.
Crucible exposed the duplication systematically.
Determinism Demonstrated, Not Assumed
We executed the same merge twice using the same corpus, cluster, and temperature-zero configuration.
Nemotron generated differently named and differently worded skills.
The deterministic gate produced identical verdicts, reasons, and finding references.
The generated content varied.
The acceptance decision did not.
The experiment produced inspectable evidence rather than relying on architectural assertions.
An Auditor That Audits Itself
Crucible's mutation laboratory deliberately introduces defects into test fixtures.
Results:
6 mutations detected. 2 correctly classified as out of scope.
We also identified a misleading claim in our own demonstration interface.
It reported that all 28 checks had passed, even though seven checks had no applicable input to examine.
We corrected the artifact to report those outcomes explicitly as ABSTAINED, identifying the missing inputs rather than silently treating non-execution as success.
That distinction is essential.
An auditor should be held to the same evidentiary standards it applies to everything else.
Tested Across 2,806 Skills
Crucible has been exercised against 2,806 skills across five corpora.
Four corpora originated from independent third parties unfamiliar with the project.
Two were frozen and held out specifically to prevent detector tuning against their contents.
This evaluation design reduces the risk of optimizing the detector merely to perform well on familiar examples.
What Makes Crucible Different
Deterministic by design. Sealed decision paths use exact arithmetic rather than floating-point approximations. Identical inputs reproduce identical SHA-256 seals across processes and four different hash seeds.
The model never decides. Nemotron proposes merges, repairs, and confirmations. Deterministic gates accept or reject them. The distinction is explicit in generated artifacts.
Abstention is a first-class outcome. Crucible distinguishes PASS, WARN, and FAIL while explicitly reporting ABSTAIN when available evidence does not support a decision. Missing evidence never silently becomes success.
Evaluation integrity is enforced in code. Corpora are frozen by commit and manifest digest. Held-out corpora record no tuning consumption; tuning corpora record what consumed them. Corpus selection uses metadata established before accuracy results are available.
Evidence remains traceable. Findings are connected to source-addressable records, reproducible decisions, and cryptographic seals.
This is not a prompt wrapped around an LLM call.
It is an auditing architecture designed so that trusting the result does not require trusting the model.
What Comes Next
Crucible's mission is deliberately precise:
Turn the methodologies installed into AI agents into auditable objects, with evidence attached to their claims.
We have also built the infrastructure for independent precision and false-negative evaluation.
The evaluation materials are sealed, unlabelled, and prepared for review without exposing the expected answers to the detector.
That independent review is the next stage.
What exists today already demonstrates the value of the approach:
- Real duplication detected and verified across external skill collections.
- Deterministic decision-making demonstrated through live Nemotron experiments.
- Adversarial mutation testing with explicit abstention.
- Reproducible evidence records and cryptographic seals.
- An auditing process capable of identifying misleading claims in its own outputs.
Crucible will continue beyond this hackathon.
Agent methodologies are infrastructure now. It's time we audited them like infrastructure.
Technical Overview
| Category | Implementation |
|---|---|
| AI model | NVIDIA Nemotron 3 Super 120B |
| Inference provider | Nebius Token Factory |
| Model responsibility | Proposal generation and narration; never final decision authority |
| Skills analyzed | 2,806 across five corpora |
| External corpora | Four independent corpora; two frozen, sealed held-out sets |
| Automated tests | 963 |
| Architecture decision records | 27 ADRs |
| Adversarial reviews | 39 |
| Evidence records | 42 |
| Determinism | Reproducible SHA-256 seals across processes and hash seeds |
| Numerical policy | No floating-point arithmetic in sealed decision paths |
| License | Apache-2.0 |
Crucible — Audit the methodology. Verify the evidence. Trust the process, not the model.
Log in or sign up for Devpost to join the conversation.