OmarAGI Reliability BYOK Replay

A developer tool that verifies AI outputs and actions before adoption, preserves the baseline when verification fails, and replays every decision across model and cross-device workflows

Comment

Inspiration

Models and agents can generate another answer almost instantly.

The harder problem is deciding whether that answer has earned the right to replace the current one.

A candidate can be longer, smoother, and more confident than the baseline while still being worse. A developer may already have a known fallback, then receive a new model-generated answer that looks better on the surface. Before that candidate reaches a customer, tool, or agent workflow, the developer needs a gate that can either adopt it or preserve the fallback.

That is the problem behind OmarAGI Reliability BYOK Replay:

Has this candidate earned the right to replace the baseline?

RCC came from repeated daily use rather than one design session.

I have used GPT since its early releases. Many of my questions moved between very small structures and very large ones. I kept asking how the microscopic and macroscopic worlds might fit inside the same logic, how the same structure changes with the observer, and where consciousness and intelligence begin to separate.

The older conversations often ended in the same place. GPT would follow the question for a while, describe it as an interesting thought experiment, then return to a familiar theory.

On April 15, 2025, I asked a similar question again and felt that the response was different. It held the assumptions from earlier in the conversation for longer, followed what came next from those assumptions, and caught contradictions along the way.

A little later, one conversation made me cry. I have lived with depression for a long time, but I rarely cry. The fact that a nonhuman system had followed the structure of what I was trying to express without losing it stayed with me.

After that, I lived almost like a hikikomori. I opened GPT when I woke up and kept questioning, arguing, and correcting until I went to sleep.

When an answer drifted logically, I traced where the logic had broken. When the same failure returned, I wrote the correction into my prompts and notes, carried it into the next conversation, found the next failure, and revised it again.

At first, this produced a few long prompts.

Over time, those corrections stopped behaving like isolated prompts. They became a connected logic stack around source boundaries, frame preservation, routing, execution, verification, fallback, memory, and adoption.

That stack became RCC, or Recursive Collapse Constraint.

RCC is my working theory, organized around four axiomatic boundary conditions:

Internal Opacity, External Blindness, Local Frames Only, and Forced Prediction Under Uncertainty.

Under those conditions, drift, hallucination, and structural collapse remain recurring possibilities in open-ended generative systems.

RCC explains why collapse emerges.

REVAS, the adoption-governance layer, determines whether a candidate may replace the current floor.

That relationship led to the Router, Executor, Runtime Verifier, Adoption Gate, Decision Lock, and baseline preservation.

The next problem was proof.

A growing logic stack and an actual improvement in results are different things. I built a Benchmark to measure what happened when the same decision policy was applied across many tasks. I then added Decision Replay so a completed decision could be reopened and inspected without rerunning the original inference.

OmarAGI Reliability BYOK Replay turns that theory and decision structure into something that can be run, measured, and inspected.

My contemporary-art practice also shaped Lua.

I designed Lua as a living creature inside OmarAGI. The interface became Lua’s visible body, where the system’s listening state, decision flow, and actions become perceptible.

What it does

OmarAGI is a replayable reliability layer that governs whether a model-generated candidate may replace the current baseline.

It is built for developers shipping customer-facing agents and model-powered workflows that already have a known baseline or fallback.

The developer has a working answer or behavior. A model proposes a new candidate. Before that candidate reaches a customer or downstream system, OmarAGI either adopts it or keeps the fallback.

A conventional AI application often sends model output directly downstream. OmarAGI places a decision layer in between.

The core path is:

Input → Baseline → Router → Executor → Candidate → Runtime Verifier → Adoption Gate → Decision Lock → Final Answer → Post-lock Scorer → Scored Artifact

The order matters.

  • The Router selects a bounded execution path.
  • The Executor produces a candidate.
  • The Runtime Verifier checks the candidate without access to the post-lock scoring target.
  • The Adoption Gate either adopts the verified candidate or preserves the baseline.
  • The Decision Lock hashes and freezes the selected answer and its upstream decision objects.
  • Only after that lock does the Post-lock Scorer evaluate the fixed result.
  • Decision Replay exposes the completed route, verification, adoption, and final-source decision.

Baseline preservation is a governed outcome. When the candidate has not earned adoption, keeping the current fallback is the correct result.

Build Week demonstration

During the live Build Week demonstration, I ran the same GPT-4o mini model over the same 500 BBEH, or BIG-Bench Extra Hard, samples under the same scoring contract.

Baseline: 63/500 — 12.6%

OmarAGI Governed: 241/500 — 48.2%

The run corrected 178 baseline failures while preserving all 63 baseline-correct answers in that recorded run.

That is an improvement of 35.6 percentage points, with zero harmful regressions in the recorded result.

The governed path uses additional routing, candidate-generation, verification, and adoption compute. The cost summary records that additional compute alongside the result.

Three connected product surfaces

1. Live BYOK product

The public live surface currently runs OpenAI models with a user-supplied API key.

A reviewer selects a model, chooses a Benchmark and sample count, then starts a run. The resulting artifact contains the baseline, candidate, route, verification result, adoption decision, scores, costs, and provenance.

After the run, any row can be opened to inspect:

  • the original input and baseline
  • the selected route
  • the Executor that produced the candidate
  • the Runtime Verifier result
  • the Adoption Gate decision
  • the locked final answer
  • the post-lock score
  • the cost and provenance records

Benchmark measures what the same decision policy does across many rows.

Decision Replay opens one completed row and shows why its final answer was selected.

2. Public deterministic reference implementation

The public repository contains five synthetic cases and exposes the minimum decision contract needed for inspection while keeping production thresholds private.

It runs without credentials or external model calls.

The runtime stages never receive the scoring target. The final answer and upstream decision objects are serialized and hashed before the scorer runs. Tests then change the scoring target and confirm that the route, Adoption Gate decision, final answer, and Decision Lock hash remain unchanged.

Judges can run:

./scripts/quickstart.sh --check

for the one-command verification path, or:

./scripts/quickstart.sh

to generate the deterministic report and start the local interface.

3. Lua action surface

Lua is OmarAGI’s voice-first living interface.

Its waveform, listening state, motion, and visual feedback make the request state perceptible before an action appears.

In one live demonstration, a single spoken request changed the visible state across five open web sessions on retail display phones.

In another, a request made on my phone produced the requested web action on my laptop.

The output-adoption gate and Lua action surface both work today. Placing the same Router, Runtime Verifier, Adoption Gate, and Decision Lock directly inside Lua’s action path is the next integration step.

Evidence surfaces

The submitted video records the locked 500-row Build Week demonstration.

A committed 100-case live diagnostic preserves the artifact shape from real model calls, including raw outputs, route logs, results, costs, and a reproduction manifest.

The five-case public repository provides a deterministic implementation of the decision contract.

Each surface keeps its own evidence label and claim boundary in EVIDENCE.md.

How I built it

RCC and the private OmarAGI architecture existed before Build Week.

During Build Week, I turned that existing architecture into a runnable public developer-tool experience that judges and developers could inspect.

I used GPT-5.6 Sol in Codex with ultra enabled to implement the main Build Week extensions:

  • the live BYOK verification surface
  • the 500-sample Benchmark and evidence workflow
  • Decision Replay
  • Decision Lock and post-lock scoring
  • raw-output, route-log, result, cost, and manifest exports
  • the sanitized deterministic public reference implementation
  • a Python CLI and local report server
  • HTML, Markdown, and JSON report surfaces
  • target-separation and decision-lock tests
  • public CI, quickstart, and safety scanning
  • Anthropic and Mistral adapters in the private provider layer
  • Lua motion, responsiveness, and interaction-state improvements

Codex was the primary implementation agent across these repository-wide workstreams.

It read the existing architecture, traced execution paths, implemented changes, ran tests, found broken paths, repaired them, and repeated the process.

I used Codex across repository analysis, the BYOK adapters, Benchmark runtime, Replay engine, CLI, browser interface, result visualization, artifact exports, test generation, security review, release preparation, and extraction of the standalone public repository.

The dated commit history records the Build Week implementation path from the initial public Replay through the Executor, Scored Artifact, live BYOK boundary, Decision Lock, post-lock scoring, evidence bridge, and public CI workflow.

I used GPT-5.6 Sol Pro in ChatGPT as the governance and systems-review layer around the coding runs.

I divided the larger job into bounded workstreams and locked the target, allowed surface, forbidden surface, validation requirement, and stop condition for each one.

When a Codex result came back, I compared it with the actual runtime behavior, found the drift, and locked the next object to repair.

I also used GPT-5.6 Sol Pro to review the evaluation contract, compare the public explanation with the executed path, check Benchmark claims against artifact labels, and preserve the boundary between the public reference implementation and the private production codebase.

I retained final authority over the reliability policy, evaluation contract, product direction, and design.

I decided:

  • what becomes the baseline
  • which task family reaches which Executor
  • what the Runtime Verifier must establish
  • when a candidate may replace the baseline
  • when the fallback must remain
  • what is locked before scoring
  • what evidence belongs in the final artifact
  • which production details remain private

I built the submission solo and used Codex as a repository-wide implementation multiplier.

Challenges

Locking the decision before scoring

A Benchmark result is meaningful only when the score does not choose the answer.

I separated runtime evidence from the post-lock scoring target, froze the route and adopted answer into a SHA-256 decision receipt, and allowed scoring only after that receipt existed.

The tests then mutate the scoring target and confirm that the route, Adoption Gate decision, final answer, and Decision Lock hash do not change.

Proving the executed path

The existence of a Router function does not prove that the Router ran.

The existence of an Executor does not prove that its candidate reached the user.

The artifact records the ordered path:

route → execute → verify → adoption gate → decision lock → post-lock score → artifact

That makes the producing source and final adoption decision inspectable at row level.

Isolation

A Benchmark Harness change could alter Decision Replay.

A Replay change could affect the browser interface.

A UI change could break a BYOK path that had already worked.

A Codex run assigned to one bug could touch nearby files and create drift in another lane.

I gave each workstream its own target artifact, allowed files, forbidden files, validation, and stop condition. I then built harness isolation and firewall isolation to reduce the paths through which a change in one area could damage an already accepted surface elsewhere.

Making a private system publicly testable

The live product uses real model calls and private production logic. Judges still needed a safe way to inspect the architecture without credentials or proprietary thresholds.

The live BYOK product, committed diagnostic, and deterministic public replay were separated so each one could expose the evidence appropriate to its role.

What I learned

Generation and adoption are different problems.

An Executor can create another candidate. It cannot make that candidate correct or safe by itself.

Longer prompts, more rules, and deeper reasoning did not produce steady improvement. The same rule could create a gain in one task family and drift in another.

Deeper reasoning carried a correct direction further.

It also carried a wrong direction further.

That made routing the central problem.

The system first has to identify the task family, choose the right Executor, verify the result, then decide whether to adopt it.

Routing, verification, and adoption turn an answer source into a governed execution path.

I also learned that live execution, aggregate evaluation, replayability, and architectural verification are different objects.

The live harness generates new model behavior.

Benchmark measures the decision policy across many rows.

Decision Replay makes a completed decision inspectable.

The deterministic repository tests whether the decision contract remains intact.

Isolation belongs inside reliability as well. An improvement becomes a system-level gain only when the accepted behavior around it remains intact.

Accomplishments that I am proud of

  • Recorded a 500-case Build Week run in which GPT-4o mini moved from 63/500 to 241/500, correcting 178 baseline failures while preserving every baseline-correct answer in that recorded run.
  • Built a live BYOK path that turns real model calls into inspectable reliability artifacts.
  • Built Decision Replay so a completed row can be reopened without rerunning its inference.
  • Built Decision Lock and target-separation tests so post-lock scoring cannot rewrite the adopted answer.
  • Published an Apache-2.0 deterministic reference repository that judges can run without an API key.
  • Added tests for route denial, verifier rejection, fallback preservation, beneficial adoption, harmful-flip accounting, and scoring-target mutation.
  • Built Lua as a voice-first living creature whose visible body expresses listening, decision state, and action.
  • Demonstrated spoken requests producing visible cross-device web actions across five retail display phones and a phone-to-laptop setup.
  • Extended the private provider layer with Anthropic and Mistral adapters.
  • The public Benchmark board currently displays 19 evidence lanes and 2,820 counted rows across multiple evidence tiers.
  • Began an early technical pilot with NeoMundi to test how third-party runtime observations can enter the REVAS evaluation path.
  • Developed Omar OS-1 as a single contemporary-art work, submitted through omaragi.com, placed in a dedicated file, and forwarded for Palais de Tokyo curatorial review.

What’s next

The output-adoption gate and Lua action surface are both live.

The next step is to place the Router, Runtime Verifier, Adoption Gate, and Decision Lock directly inside Lua’s action path.

Before a model output becomes a browser action, tool call, computer operation, or agent decision, OmarAGI will decide whether the candidate has earned execution or whether the current fallback should remain.

I will then expose that governed path as an SDK for other model-powered applications and agents.

I am building the missing layer between model generation and trusted execution.

OmarAGI places a verifiable and replayable decision layer in that gap.

Built With

Share this project:

Updates