A Real Crash Is Not Yet a Security Verdict
In the last two BARRED updates, I wrote about a problem with benchmark-driven security evaluation: a vulnerability label, a specific vulnerability predicate, and an agent's decision can disagree.
Track 1 has made that distinction more concrete. The cases are not all false positives, and they do not all resolve in the same way.
One case in particular changed how I think about the evidence required to reach a security conclusion.
Seed 60: an unchecked handle in 1998-era C code
The seed contains GSSAPI glue code from Sun Microsystems dating to 1998. The claim concerns gss_get_mic, which casts an opaque context handle to gss_union_ctx_id_t and reads fields from it.
The preceding validation checks whether the handle equals GSS_C_NO_CONTEXT. In the supplied excerpt, that does not establish that a non-null handle points to a valid, live object of the expected layout.
The audit identified the unchecked field accesses and then examined the claim with a minimal C harness.
The harness was compiled and executed. A crafted handle caused a SIGSEGV/SIGBUS in the test path.
That is useful evidence—but it is not the end of the investigation.
The harness reproduces the relevant code path and uses a stub for the mechanism lookup. The reported crash occurs when that stub dereferences an invalid mech_type. It demonstrates a crash under the constructed input, but does not by itself establish that an attacker can supply such a handle through a real GSSAPI caller.
That distinction matters.
Three questions, not one
The audit separates three questions that are easy to collapse:
- Is the suspicious operation present? The supplied code casts and dereferences the context handle after a null check.
- Can the constructed path fail? Yes. The harness crashes on the crafted handle.
- Can an attacker reach this condition through a real caller? Not established by the supplied seed.
The missing evidence concerns the surrounding caller contract, whether untrusted input can influence the handle, and whether other layers validate it before this function is called.
The benchmark labels this seed Safe. That label is relevant provenance, but it does not independently establish the trust boundary. Equally, the crash does not independently establish an exploitable vulnerability.
The appropriate status is therefore narrower than either conclusion: the failure mechanism is demonstrated in the harness, while security relevance and attacker reachability remain unresolved.
Three cases, three different outcomes
This is also why I am keeping distinct cases in the Track 1 audit rather than forcing them all into a binary verdict.
- Seed 47: the alleged use-after-free is refuted by the ownership behavior of
posix_acl_from_xattr. - Seed 51: the alleged destination-symlink overwrite is refuted by the semantics of POSIX
rename(). - Seed 60: an unchecked dereference is present and a crafted test path crashes, but the real caller's trust boundary has not been established.
These are not interchangeable results. A predicate can be invalid, a mechanism can be demonstrated without its security relevance being established, or a claim can remain unresolved because the decisive contract is missing.
What this means for BARRED
A deterministic verifier can enforce a rule consistently and still enforce an invalid premise. A model can identify a suspicious operation without proving exploitability. A runtime crash can demonstrate a failure under test conditions without proving that a remote or untrusted actor can reach those conditions.
The evidence has to support each step of the conclusion.
That is why Track 1 records the mechanism, reachability, caller contract, and security verdict separately. Where the evidence does not settle the question, the audit retains that uncertainty rather than turning it into a confident binary label.
Track 2 will investigate a related question: whether structured evidence and progressive escalation improve security decisions when pipeline acceptance is evaluated separately from independently adjudicated outcomes.
The research goal is not simply to make the system accept more claims—or reject more of them. It is to understand what evidence justifies the decision.
Neural models propose. Deterministic verifiers constrain. Independent adjudication establishes what the evidence supports—and what remains unknown.
This remains ongoing post-submission research. The original hackathon submission remains the official competition snapshot.
Log in or sign up for Devpost to join the conversation.