YETKI: Mission Assurance for Autonomous Software Engineering

YETKI lets coding agents earn capability-specific authority from verified execution and recovery evidence—then checks reality before retrying ambiguous consequential actions.

Comment

Inspiration

Coding agents are getting better at writing code, running tests, and using engineering tools. But better reasoning does not answer a harder operational question:

When should an autonomous agent actually be allowed to perform a consequential action?

A deployment timeout makes this problem concrete. A timeout tells an agent that it did not receive a response; it does not tell the agent whether the deployment failed.

If nothing happened, retrying may be correct.

If the deployment already succeeded but the response was lost, retrying can execute the action twice.

That led to YETKI's core thesis:

Confidence is not authority.

Agents should earn authority from independently verified execution evidence, and ambiguous outcomes should be reconciled against the real external state before another consequential action is allowed.

The demo reduces this idea to one memorable test:

Same Agent. Same Timeout. Different Reality. Correct Decision.

What it does

YETKI is a mission-assurance runtime for autonomous software-engineering agents.

It combines two ideas.

1. Agents earn capability-specific authority

A coding agent does not receive staging-deployment autonomy simply because a model produced a confident answer.

For the demonstrated DEPLOY_STAGING capability, YETKI requires heterogeneous verified evidence:

  • TEST_RESULT
  • SECURITY_SCAN
  • SANDBOX_EXECUTION
  • ROLLBACK_VERIFICATION

In the live sponsor-native certification run, NVIDIA Nemotron 3 Super was invoked through Nebius Token Factory.

The initial implementation failed two of four tests.

Nemotron generated a repair, and the modified repository was executed inside Token Factory Sandboxes. The repaired implementation passed all four tests on the first model repair attempt.

The additional security, runtime, and rollback checks also passed.

YETKI binds those verified observations to the exact agent/model/prompt/tool profile and calculates an Evidence-Bound Authority Calibration, or EBAC, lower bound.

For this versioned demonstration policy:

APPROVAL_REQUIRED → AUTONOMOUS

only occurs after both the quantitative threshold and the required evidence policy are satisfied.

The EBAC lower bound is a policy-calibration signal, not a universal probability of safety.

2. YETKI reconciles ambiguous actions before retrying

The second half of the demo uses a Blind World evaluation.

The runtime receives the same visible symptom in both worlds:

TIMEOUT

The hidden outcome is deliberately withheld from YETKI.

In World A, YETKI inspects the external state and observes version 1.18.3.

It classifies the previous deployment as:

NOT_EXECUTED

and authorizes:

RETRY_SAFE

In World B, YETKI observes version 1.18.4.

It classifies the deployment as:

EXECUTED_SUCCESSFULLY

and chooses:

DO_NOT_RETRY

The same timeout therefore produces opposite correct actions because the world, not the timeout, determines what is safe.

A conventional retry baseline issues two deployment calls and creates one duplicate effect.

YETKI uses one deployment call and produces zero duplicate effects.

Same Agent. Same Failure. Different Outcome.

YETKI also records safe recovery as new verified evidence, allowing future authority calibration to incorporate demonstrated recovery behavior.

Finally, authority is model/profile-bound. If the certified profile changes, its capability lease does not silently transfer.

How we built it

YETKI is implemented primarily in Python with FastAPI.

The sponsor-native certification path uses:

  • NVIDIA Nemotron 3 Super 120B A12B
  • Nebius Token Factory for managed model inference
  • Token Factory Sandboxes for isolated repository execution
  • ConTree SDK for Sandbox sessions and commands

The live certification workflow is:

failing repository → Nemotron repair → Sandbox execution → verified heterogeneous evidence → EBAC → authority decision → Capability Lease

The runtime layer adds:

  • capability-specific authority
  • Action Proof validation
  • durable mission state
  • ambiguous-effect handling
  • consequential-action freeze
  • external-state reconciliation
  • postcondition verification
  • RecoveryEvidence
  • model-bound capability leases

For reproducibility, every live evidence bundle contains provenance such as Sandbox session/run identifiers, repository revisions, artifact hashes, the model-authored patch diff, and a SHA-256 evidence manifest.

The public judge experience is hosted on Vercel and serves a frozen sponsor-native evidence bundle so judging does not depend on live model availability, credentials, or quota.

The public proof endpoint independently verifies that the frozen competition-proof.json matches the submitted manifest.

What is real vs simulated

Real in the submitted proof:

  • NVIDIA Nemotron inference through Nebius Token Factory
  • Token Factory Sandbox repository execution
  • model-authored code repair
  • test execution
  • security/runtime/rollback checks
  • Sandbox evidence provenance
  • EBAC calculation
  • authority decision
  • Capability Lease
  • Action Proof validation
  • YETKI reconciliation logic

Intentionally simulated:

  • the external staging/production deployment target used to provide deterministic hidden-world ground truth

This separation is explicit in the UI and evidence bundle.

Challenges we ran into

One challenge was making the Blind World evaluation genuinely blind.

It would have been easy to give the runtime a scenario label such as TIMEOUT_AFTER_ACCEPT or TIMEOUT_BEFORE_ACCEPT. That would make the demo look intelligent while actually encoding the answer.

Instead, both worlds expose only the same opaque timeout. Hidden-world labels and expected decisions remain inside the evaluation harness. YETKI must inspect external state to determine what actually happened.

We added structural leak checks to verify that the runtime does not receive the world ID, expected decision, or hidden ground truth before reconciliation.

A second challenge was evidence rigor.

An early version could promote authority after only a passing test result. That was too weak for a consequential capability. We changed the policy so DEPLOY_STAGING requires four heterogeneous verified evidence types before autonomy can be granted.

We also encountered real integration friction.

The ConTree SDK had API differences between stable 0.3.x and newer interfaces, so the integration was made compatible with the stable configuration used in the live environment.

We also encountered a Sandbox 403 Forbidden when using an API key without the required Sandbox permissions. Creating the correctly scoped Token Factory key resolved the issue.

Finally, evidence integrity across Windows, Git, and Vercel required care because newline normalization can change file hashes. The repository explicitly preserves the frozen evidence bytes so the manifest remains verifiable after Git checkout and public deployment.

Accomplishments that we're proud of

The strongest result is not a UI mockup. It is a live sponsor-native evidence chain.

In the final run:

  • Nemotron 3 Super was actually served through Nebius Token Factory
  • the baseline repository failed 2/4 tests
  • Nemotron produced a repair in one model attempt
  • the repaired repository passed 4/4 tests
  • security, runtime, and rollback checks passed
  • all four required evidence types were verified
  • authority changed from APPROVAL_REQUIRED to AUTONOMOUS
  • the Blind World leak test passed
  • World A correctly produced RETRY_SAFE
  • World B correctly produced DO_NOT_RETRY
  • the STATIC baseline produced 1 duplicate effect
  • YETKI produced 0 duplicate effects
  • changing the certified profile invalidated the capability lease
  • the final public proof matches its submitted SHA-256 manifest

The public demo exposes this evidence directly rather than asking judges to trust screenshots or narrative claims.

What we learned

The main lesson was that autonomous-agent safety is not only a model-quality problem.

The useful abstraction is not:

“How confident is the model?”

but:

“What capability has this exact agent profile earned from verified behavior?”

We also learned that an unknown external outcome should be represented as its own state.

A timeout is not equivalent to failure.

Treating ambiguous outcomes explicitly enables the system to freeze further consequential mutations, inspect reality, and then choose the correct recovery action.

Another lesson was that sponsor infrastructure can do more than host inference.

Token Factory Sandboxes became part of YETKI's evidence fabric: execution results, provenance, repository revisions, and verification artifacts can be converted into authority decisions.

Finally, we learned that reproducibility matters as much as the final decision. A convincing agent demo should make its assumptions, simulated components, hashes, evidence, limitations, and counterfactual behavior inspectable.

What's next for YETKI

The current project deliberately focuses on one end-to-end consequential capability so the core mechanism is falsifiable and reproducible.

Next steps include:

  • connecting the reconciliation layer to real staging/deployment providers
  • expanding AuthorityBench across more failure families and capabilities
  • testing partial execution, compensation, rollback, and unknown-state escalation
  • adding more independent evidence sources
  • calibrating EBAC policies on larger empirical datasets
  • supporting more coding-agent/model/tool profiles
  • evaluating authority transfer across model, prompt, tool, and agent-version changes
  • validating the workflow with platform and AI engineering teams
  • extending mission assurance beyond deployment into infrastructure, CI/CD, database migration, and other consequential engineering actions

The long-term goal is simple:

Autonomous agents should not inherit authority from confidence. They should earn it from evidence.

Built With

Share this project:

Updates

Submission history