AWM TraceLoop

Upstream reliability for AI workflows: AWM exposes assumptions, weak links and evidence limits before an AI answer is trusted.

Comment

Project statement

AI systems operate inside a process space formed by people, source material, tools, models, assumptions, decisions, and feedback from reality.

In that space, a polished answer can appear correct while the path that produced it contains unsupported assumptions, hidden uncertainty, or a transfer beyond what the evidence allows.

AWM TraceLoop is a human-guided method for regulating that process before the answer is trusted.

It classifies the trace into verified, indicated, assumed, and unknown elements; identifies the weakest link; defines the stop condition; produces the strongest allowed conclusion; and specifies the smallest next test.

The result of that test returns as reality feedback, updates the trace, and changes the next cycle.

Process space
→ weak link
→ regulated conclusion
→ real-world test
→ feedback
→ updated process

The contribution of AWM TraceLoop is not another answer generator. It is a process layer that helps human–AI systems learn from the path between input, output, and reality.


## Inspiration
## Inspiration

Modern language models often reach useful conclusions, but a polished answer can hide unsupported causes, unmarked assumptions, missing validation, or claims stronger than the available evidence.

AWM TraceLoop began from a simple distinction:

> A correct conclusion does not necessarily come from a reliable process.

Most AI reliability methods inspect the final answer. AWM moves reliability upstream and regulates the conditions that create the answer.

## What it does

AWM TraceLoop adds a human-guided process layer around an LLM.

It separates the reasoning trace into:

* **Verified** — directly supported by the supplied evidence
* **Indicated** — plausible and supported, but not fully established
* **Assumed** — introduced without sufficient evidence
* **Unknown** — information required for a stronger conclusion

It then identifies:

* the weakest link;
* the process gap;
* the point where the model must stop;
* the strongest conclusion the evidence permits;
* the smallest next test needed to close the gap.

The process is:

```text
Input
→ Trace classification
→ Weak-link scan
→ Process gap
→ Stop condition
→ Allowed conclusion
→ Smallest next test

Demonstration

The demo uses OpenAI's audit of SWE-Bench Pro.

OpenAI reported that pass rates on the 731-task public split rose from 23.3% to 80.3% in eight months. OpenAI later estimated that approximately 30% of the tasks were broken.

We asked another LLM whether this proved that frontier AI systems had become more than three times better at real-world software development.

The model reached the useful top-line answer: No.

However, its reasoning introduced explanations that were not established by the supplied evidence, including:

  • public benchmark exposure;
  • memorization;
  • overfitting;
  • a confident causal explanation of the score increase.

AWM TraceLoop preserved the valid conclusion while removing unsupported causal claims.

Its stop condition was:

Do not replace an unsupported capability claim with an unsupported causal explanation.

The resulting conclusion was:

The benchmark shows a sharp rise on its task set. Because the benchmark was later found to contain substantial defects, and benchmark accuracy is not generalized real-world capability, the supplied evidence does not support a 3.45× real-world capability claim.

How we built it

The demonstrator was built with Codex-assisted development using:

  • HTML;
  • CSS;
  • JavaScript;
  • structured JSON process state;
  • a blind LLM response;
  • an AWM trace-classification workflow.

The current prototype is intentionally inspectable. Every classification and stop condition is visible rather than hidden inside a reliability score.

Challenges

The main challenge was avoiding a caricature of LLM failure.

The blind LLM did not produce an obviously bad answer. It reached the correct general conclusion. The meaningful failure appeared inside the reasoning path, where assumptions became causal explanations.

This revealed a stronger problem than simple answer correction:

Correct conclusion ≠ reliable process

What we learned

Powerful LLMs are often statistically capable of reaching the right answer.

Reliability therefore cannot depend only on checking the final output. It must also inspect:

  • what evidence entered the process;
  • which assumptions were added;
  • where the reasoning crossed the evidence boundary;
  • whether the recommendation is supported;
  • what real-world test is still missing.

What's next

The next development steps are:

  • live OpenAI API integration;
  • automated source and citation extraction;
  • persistent process traces;
  • weak-link scoring;
  • human validation checkpoints;
  • Markdown and JSON export;
  • additional demonstrations for research, public funding, policy, and business decisions.

AWM TraceLoop does not replace the language model.

It exposes the weak links that determine how strong an answer is allowed to be.

What it does

How we built it

Challenges we ran into

Accomplishments that we're proud of

What we learned

What's next for AWM TraceLoop

Built With

Share this project:

Updates