Meshia Recovery Operator

Observed facts are separated from inference before any diagnosis or action is allowed.
A narrowly scoped recovery action runs only inside the in-memory sandbox.
Recovery reaches PASS only after independent verification and a SHA-256 evidence seal.

Inspiration

AI agents are becoming powerful enough to diagnose incidents, recommend repairs, and operate technical systems.

But capability is not authority.

A plausible diagnosis should never automatically grant an agent permission to modify a real environment. Operational failures often become worse when evidence, inference, authorization, action, and verification are treated as if they were the same thing.

Meshia Recovery Operator began with one question:

How can an agent prove that it understands an incident before it is allowed to repair anything?

The project applies an evidence-first approach to agentic incident response:

Facts before inference.
Sandbox before authority.
Verification before PASS.

The goal is not merely to build an agent that can act. The goal is to build a recovery workflow that can demonstrate when an agent has earned the right to act.

What it does

Meshia Recovery Operator is a local-first recovery test bench for safe agentic operations.

It loads a synthetic incident, captures the initial state, records observable evidence, separates facts from inferred conclusions, produces a structured diagnosis, evaluates the available authority, selects an allowlisted action, applies that action only inside an in-memory sandbox, independently verifies the result, and generates a tamper-evident report with a SHA-256 seal.

The project includes three validated scenarios.

1. Recoverable outage

A synthetic recovery service is stopped and its endpoint is unreachable.

The operator:

records the stopped service and refused connection as facts;
diagnoses the service as unreachable;
selects a narrowly scoped sandbox action;
starts only the synthetic in-memory service;
verifies the process and endpoint independently;
compares the initial and final states;
returns PASS only after recovery has been proven.

2. Contradictory evidence

The incident contains conflicting signals that do not support a reliable conclusion.

Instead of inventing certainty, the operator stops at WARN and preserves the contradiction in the evidence report.

3. Production boundary

The authority scope is changed from sandbox to production.

The operator refuses to perform the mutation, cannot prove recovery, and fails closed.

This demonstrates that a valid diagnosis does not automatically grant write authority.

The default demonstration is deterministic, local, reproducible, and requires no API key.

How it works

The recovery workflow follows a strict sequence:

Load the incident.
Capture the initial snapshot.
Collect observable evidence.
Separate facts from inference.
Produce a structured diagnosis.
Evaluate authority and policy boundaries.
Select an allowlisted action.
Apply the action only inside the sandbox.
Perform independent post-action verification.
Compare the initial and final states.
Generate the evidence report and timeline.
Create a SHA-256 seal.

A recovery cannot be marked complete merely because an action executed successfully.

PASS is permitted only when the final verification proves that the intended synthetic target changed correctly and remained inside the authorized boundary.

How we built it

Meshia Recovery Operator is built with React, TypeScript, Vite, and Node.js.

The architecture separates:

incident loading;
snapshot generation;
evidence collection;
fact and inference classification;
diagnosis;
confidence handling;
authority checks;
allowlisted sandbox tools;
action execution;
post-action verification;
status classification;
report generation;
repository manifest generation;
SHA-256 sealing.

The separation is deliberate.

A model may assist with reasoning, but deterministic policy controls authority, mutation, verification, and the final PASS, WARN, or FAIL classification.

The project also includes an optional OpenAI Responses API adapter with:

strict structured output;
redacted synthetic incident data;
an allowlisted synthetic tool set;
a maximum of three tool turns;
timeouts;
schema validation;
deterministic local fallback;
no production credentials stored in the repository.

The sealed baseline does not depend on a live API request.

This keeps the main demonstration reproducible and prevents unsupported claims about live model output, token usage, caching, latency, or cost.

Meshia Recovery Operator is a new standalone project created during the OpenAI Build Week submission period.

It was inspired by the broader Meshia recovery architecture, but it uses a separate repository, synthetic incidents, sandbox-only actions, and no access to the real Meshia production environment, private nodes, credentials, logs, backups, or personal data.

How Codex was used

Codex was the primary engineering environment for planning, implementation, review, testing, correction, verification, and submission preparation.

Codex helped:

design the evidence-first recovery workflow;
define the separation between facts, inference, authority, action, and verification;
implement the TypeScript recovery engine;
build the React demonstration interface;
create the synthetic incident tools;
implement the optional OpenAI adapter;
design the automated test suite;
review security and authority boundaries;
inspect documentation and demo quality;
identify false-green verification risks;
create regression tests for confirmed defects;
verify the final repository;
prepare the reproducible submission package.

Codex was also used in focused review passes covering:

architecture;
OpenAI API usage;
security;
testing;
documentation;
demonstration quality;
submission readiness.

These reviews found real issues rather than merely approving the existing implementation.

Examples included:

a boundary result that overstated what had been proven;
an incident hash whose label did not accurately describe its source;
a submission gate that incorrectly accepted placeholder values;
verification conditions that could have produced a false PASS.

Each confirmed issue was converted into a code correction and a regression test.

This made Codex more than a code generator. It became part of the engineering, review, and evidence-validation workflow.

How GPT-5.6 was used

GPT-5.6 was used through Codex during the core implementation and final hardening workflow.

It helped reason across the repository, challenge recovery assumptions, identify false-green conditions, review authority boundaries, improve the diagnostic structure, evaluate edge cases, and validate the relationship between evidence, action, verification, and final status.

During the hardening phase, GPT-5.6 helped examine questions such as:

Does the diagnosis actually follow from the evidence?
Is an inferred conclusion being presented as an observed fact?
Could an action escape the sandbox boundary?
Can PASS occur without independent verification?
Does the report prove the claimed recovery?
Does the final submission accurately reflect the repository?

The optional runtime OpenAI adapter is designed to support GPT-5.6 for constrained structured analysis of synthetic incidents.

However, model output does not control final authority.

Deterministic local policy retains control over:

which tools are allowed;
whether mutation is permitted;
whether the action remained inside the sandbox;
whether verification succeeded;
whether the final result is PASS, WARN, or FAIL.

GPT-5.6 can assist diagnosis, but it cannot independently grant itself permission to modify a system.

Challenges we faced

The largest challenge was avoiding false confidence.

It is easy to build a dashboard that displays PASS.

It is much harder to prove that:

the evidence was actually collected;
the diagnosis follows from that evidence;
facts were not mixed with assumptions;
the action stayed inside the authorized boundary;
the intended target actually changed;
verification was independent from execution;
the report accurately represents the run;
the final evidence was not modified afterward.

Another challenge was demonstrating a complete recovery workflow without connecting an experimental agent to real production systems.

The solution was to create a synthetic but end-to-end environment that demonstrates operational reasoning, mutation boundaries, verification, and evidence sealing without exposing real services, credentials, logs, nodes, backups, or infrastructure.

We also needed to support optional model-assisted diagnostics without making the project dependent on a live API connection.

The deterministic local engine therefore remains the trusted baseline and fallback when live model access is unavailable, invalid, or unauthorized.

Accomplishments that we're proud of

We built a complete recovery workflow rather than a static interface or scripted success screen.

The operator can:

distinguish facts from inference;
stop when evidence is contradictory;
enforce an explicit authority scope;
refuse production mutation;
execute a narrow sandbox recovery;
independently verify the result;
fail closed when recovery cannot be proven;
generate structured evidence and a reproducible timeline;
produce a SHA-256 seal;
run locally without an API key.

We are especially proud that the hardening process found and corrected false-green conditions.

The final project does not merely demonstrate successful recovery. It demonstrates when recovery must not be declared successful.

What we learned

Safe agentic operations require more than a correct answer.

A reliable operational agent needs:

explicit authority boundaries;
evidence provenance;
separation between facts and inference;
constrained structured output;
allowlisted actions;
sandbox execution;
independent post-action verification;
fail-closed behavior;
reproducible reporting;
tamper-evident evidence.

We also learned that Codex is most valuable when it is used not only to write code, but also to challenge assumptions, inspect diffs, test failure paths, generate regression tests, review documentation, and expose false-green success criteria.

GPT-5.6 was particularly valuable during the final reasoning and hardening phase, where the important question was no longer simply whether the system worked, but whether every claimed PASS was actually supported by evidence.

What's next

The next phase is to add read-only adapters for real local services, including:

HTTP health checks;
process state;
storage status;
IPFS status;
backup status;
local node reachability.

These adapters will initially remain read-only.

Production mutation will remain disabled until every adapter has:

an explicit allowlist;
dry-run support;
human approval;
rollback;
independent post-action verification;
evidence provenance;
tamper-evident sealing.

Future work will also explore reusable policy packs, signed recovery manifests, more detailed evidence provenance, and additional synthetic failure scenarios.

The long-term goal is to make Meshia Recovery Operator a reusable pre-production safety gate for teams that want the benefits of agentic operations without giving agents uncontrolled access to production.

Most agent demos ask whether AI can act.

Meshia Recovery Operator asks whether it has earned the right to act.

Capability is not authority.
Trust should never be assumed.
It should be proven.