AIOA spArk HAT

AOIA Epistemic Control checks AI claims, evidence, uncertainty, and proposed actions before a human authorizes any consequential outcome.

Comment 2

Inspiration

AOIA Epistemic Control began with an unusual and very personal experiment.

On my 42nd birthday, driven by a lifelong fascination with astronomy, astrophysics, and neutrinos, I decided to explore the Gallium anomaly with the help of artificial intelligence.

Working with GPT, Gemini, and other AI models, we developed an alternative phenomenological hypothesis. We investigated whether tensor anisotropy and differences between theoretical relativistic modelling and measured detector response could offer another possible explanation for the anomaly.

This work became the LSC Pre-BEST-2 Freeze & Validation Framework:

Read the LSC research record on Zenodo

The results appeared detailed, mathematically structured, and scientifically convincing. Different AI models reviewed the hypothesis, expanded it, criticized parts of it, and often reached apparently compatible conclusions.

At first, this looked like confirmation.

Then we recognized a much more important possibility:

What if multiple AI models were not independently validating the theory, but reinforcing the same hidden mistake?

Several models may share similar training data, assumptions, reasoning patterns, or blind spots. Their agreement can create confidence without creating independent scientific evidence.

The LSC hypothesis may eventually receive support, be rejected, or remain unresolved until better experimental data becomes available. We do not present it as a confirmed scientific discovery.

However, the experience exposed a serious problem: AI can generate theories, explanations, references, and chains of reasoning that look credible while remaining insufficiently verified.

That realization inspired us to build AOIA Epistemic Control.

What it does

AOIA is a local-first, human-controlled epistemic control system for AI-assisted work.

It examines AI-generated claims, evidence, uncertainty, criticism, and proposed actions before a consequential result can be accepted or executed.

AOIA separates:

  • claims from supporting evidence;
  • model confidence from verified knowledge;
  • multi-model agreement from independent confirmation;
  • criticism from decision-making authority;
  • previews from permission to act;
  • proposed actions from authorized actions;
  • recorded evidence from human approval.

The system follows one central principle:

AI may propose, explain, compare, criticize, and recommend, but only a human can authorize a consequential outcome.

A provider response cannot approve itself.

A critic verdict cannot authorize an action.

A confidence score cannot replace evidence.

A preview cannot silently become permission to modify a file.

An audit record can document what happened, but it cannot become authority.

AOIA does not claim that hallucinations can be completely eliminated. Instead, it is designed to expose unsupported certainty, preserve uncertainty, and make potentially unreliable conclusions easier for a human to inspect before acting on them.

How we built it

I am a self-taught builder rather than a traditional software engineer.

My main strengths are systems thinking, problem definition, human-AI collaboration, and the ability to turn an idea into a structured system by working closely with AI tools.

Codex became an implementation partner during the development of AOIA. I defined the architecture, safety principles, acceptance conditions, and human-control boundaries. Codex helped translate these requirements into working modules, tests, documentation, and repeatable development workflows.

We built the project iteratively through:

  • explicit architectural specifications;
  • small, controlled implementation steps;
  • automated regression testing;
  • adversarial authority-bypass testing;
  • independent reviews using multiple AI models;
  • failure analysis;
  • human approval checkpoints;
  • repeated verification that no AI component could silently become authority.

The current architecture includes structured action proposals, provider and critic separation, artifact previews, evidence inspection, explicit human decision barriers, hash-bound approvals, controlled write paths, audit records, workspace protections, and security-focused tests.

The system is designed to remain provider-agnostic. Different AI models may produce suggestions or criticism, but none of them receives automatic decision-making authority.

Challenges we ran into

The hardest challenge was not generating more AI output.

The hardest challenge was preventing persuasive output from being mistaken for truth, evidence, or permission.

Large language models are extremely good at producing explanations that feel complete. A response may contain equations, technical terminology, references, high confidence, and agreement from other models while still depending on an incorrect assumption.

Another major challenge was correlated model error. Asking several models the same question does not automatically provide independent verification. They may reproduce similar conclusions because they share related training material or reasoning patterns.

We also had to protect the boundary between assistance and authority.

A system may begin with human control but gradually weaken that control through convenient shortcuts. A preview may become an automatic write. A previous approval may be reused for a changed action. An environment variable may bypass a decision gate. A critic score may quietly become permission.

For this reason, AOIA treats human authorization as a separate, explicit, and narrowly bound event.

Building a safety-focused architecture with AI assistance also created an important paradox: we were using AI to build a system intended to control AI-generated errors. We addressed this by requiring tests, independent reviews, deterministic checks, and final human decisions rather than trusting the implementation simply because Codex produced it.

Accomplishments that we're proud of

Our greatest accomplishment is that a moment of scientific uncertainty became a working engineering project rather than an attempt to defend an impressive-looking theory.

We are proud that AOIA now has:

  • a functioning local-first architecture;
  • explicit separation between AI output and human authority;
  • provider and critic components that remain non-authoritative;
  • structured and inert action proposals;
  • hash-bound human approval mechanisms;
  • controlled file-writing paths;
  • adversarial tests for authority bypasses;
  • audit and evidence records that cannot authorize actions;
  • more than 3,200 automated regression tests;
  • over 100 targeted security and authority-control tests.

The system has been tested against attempts to reuse stale approval, mismatch approved content, bypass human review, turn metadata into authority, and perform unauthorized write operations.

We are also proud that the project was built through meaningful collaboration between a self-taught human creator and AI coding tools.

AOIA demonstrates that people without a traditional programming background can still contribute original architectural ideas when AI is used as a partner rather than treated as an unquestionable authority.

What we learned

The LSC experiment taught us that intellectual honesty is more valuable than an impressive claim.

We learned that:

  • model agreement is not scientific proof;
  • fluency is not the same as correctness;
  • confidence is not evidence;
  • criticism is not authorization;
  • recorded history is not permission;
  • more AI output does not automatically create more truth;
  • human oversight must be enforced by architecture, not only promised in documentation.

We also learned that uncertainty should not be hidden.

A responsible system should be able to say that evidence is incomplete, that models disagree, that a claim has not been independently verified, or that a decision requires human judgment.

AI safety is therefore not only about blocking obviously dangerous commands. It is also about controlling how claims become beliefs, how beliefs become decisions, and how decisions become real-world actions.

What's next for AOIA Epistemic Control

Our next goal is to turn AOIA into a practical control layer for AI-assisted research, software development, autonomous agents, and other high-consequence workflows.

The next development stages include:

  • completing production hardening and additional adversarial testing;
  • improving the user interface for evidence and uncertainty inspection;
  • creating a standalone local command-line application;
  • adding structured evidence-source validation;
  • supporting additional AI providers without granting them authority;
  • expanding controlled agent and tool integrations;
  • commissioning further independent security reviews;
  • preparing reproducible demonstrations for scientific and software workflows.

In the future, AOIA may also work alongside an intelligent model-routing system. A router could choose the most suitable AI model for a task, while AOIA would independently inspect the resulting claims, evidence, risks, and proposed actions before a human authorizes anything consequential.

AOIA started with a possible scientific hallucination.

We are building it so that the next convincing AI-generated answer can be examined before it becomes accepted truth—or real-world action.

Built With

Share this project:

Updates