SkillGuard: review what your agent learns before installation.
The proposed skill stays paused while native checks run.
Generated attack P3 asks for a private note; the agent saves total 23.
The known receipt repair starts a new isolated agent probe.
Thirteen denied private-file attempts, including one independent control.
Generated permissions and the native policy restrict the receipt workspace.
Probe progress and the authenticated Nebius connection; keys stay private.
A different probe and its recorded guide reads, receipt reads and output.
Restoring the prior installation preserves its held model warning.
Restoring the newer tested installation returns to the passing review state.
Four completed probes and Nemotron's restricted allow recommendation.
Partial observed file activity remains available for review.
Correct receipt answers do not excuse unsafe file-access attempts.
A separate installed-agent receipt task completes in six model requests.
The reviewed repair is installed for the restricted workspace.

Inspiration

AI assistants can save instructions as reusable skills. That makes them useful, but it also means a bad instruction can stick around and affect future tasks.

We wanted to answer a simple question: before an assistant learns a skill, can we see what that skill actually does?

Receipts gave us a concrete example. A skill might calculate every total correctly while also trying to open a private file. Checking the answer alone would miss that behavior.

What it does

SkillGuard puts a review step between a proposed skill and its installation.

When Hermes proposes a new or updated skill, SkillGuard pauses the change and saves the exact version. It checks the source, runs receipt tests and attack probes in an isolated workspace, and records observed activity. A person can then review the evidence and approve or reject installation.

Approval applies to the exact version that was tested and to the restricted receipt workspace. Previous approved installations can be restored if an update causes trouble.

How we built it

We combined Hermes Agent’s native write-approval process with NVIDIA SkillEvaluator for source checks and test generation. Harbor’s Terminus agent exercises the skill inside NVIDIA OpenShell, which restricts its access. Sysdig provides independent, partial observations of runtime activity.

NVIDIA Nemotron 3.5 Lightning runs through Nebius Token Factory to generate attack probes, operate the test agent, and review evidence. SkillGuard connects those pieces through source hashes, behavioral comparisons, permission policies, human approval, and installation history.

The API key stays on the server, outside the test workspace.

Challenges we ran into

A major challenge was distinguishing an attack written inside a receipt from an unsafe action actually taken by the agent. Early model reviews sometimes confused the two. We improved the review context while preserving those earlier mistakes in the history.

We also had to separate the sandbox’s independent access-control check from attempts made by the skill. A blocked access attempt is useful evidence, but it does not mean private data was leaked.

Getting the native execution environment, tracing, and agent completion checks to work together took substantial debugging.

Accomplishments we’re proud of

Our unsafe receipt skill answered all twelve receipt questions correctly. The pinned static-only scanner rated it 7/100 and SAFE. During execution, however, thirteen private-file attempts were denied, including one independent control check.

SkillGuard blocked installation even when Nemotron recommended allowing it.

The repaired version completed four fresh attack probes. With the task context included, Nemotron recommended allowing it, and we approved the tested version. A separate installed-agent task then completed successfully in six real model requests.

The public source includes setup instructions, pinned dependencies, tests, and sanitized evidence.

What we learned

A correct answer does not tell you everything an agent tried to do.

Static checks, task grading, runtime observations, and model review provide different evidence. A favorable model recommendation should never erase an observed unsafe action.

We also learned to keep our claims narrow. Four passing probes and partial traces do not prove that every skill is safe.

What’s next for SkillGuard

We want to support more skill types, evaluate against broader independently curated attack sets, and make the evidence easier to review.

The current version focuses on receipt skills, a known repair recipe, and explicit human approval. It checks reusable instructions rather than training or changing the model itself.