Stale AI

Stale AI detects when an AI agent becomes outdated because its source docs, APIs, or policies changed, then generates tests and proposes a fix

Comment

Inspiration

Stale AI started when I learned that Polycystic Ovary Syndrome, commonly called PCOS, had been renamed Polyendocrine Metabolic Ovarian Syndrome, or PMOS.

I was working on a women’s health project and asked an AI tool about the condition. It continued calling it PCOS and did not recognize that the official name had changed to PMOS.

I only noticed the mistake because I was already researching the topic.

That made me wonder:

What happens when an AI system becomes outdated and nobody using it knows enough to notice?

A message saying “AI can make mistakes” does not solve this problem. It places the responsibility on people to catch errors themselves.

Developers also have no simple way to know:

  1. What information changed
  2. Which parts of their application are affected
  3. Whether their agent is now giving outdated answers
  4. How to test and repair the problem

That became the idea behind Stale AI.

What it does

Stale AI detects when an AI agent becomes outdated because a source it depends on has changed.

These sources could include:

  1. API documentation
  2. Company policies
  3. Medical terminology
  4. Product instructions
  5. Compliance requirements
  6. Internal procedures

A developer provides an older source and its updated version. Stale AI then:

  1. Identifies the meaningful changes between the sources
  2. Scans the repository for outdated references
  3. Generates regression evaluations from the updated information
  4. Runs those evaluations against the agent
  5. Shows whether the agent still produces stale answers
  6. Proposes a repair
  7. Runs the evaluations again to verify the repair

If the old source is (S_{\text{old}}) and the new source is (S_{\text{new}}), Stale AI identifies:

[ \Delta S = S_{\text{new}} - S_{\text{old}} ]

It then converts the meaningful information in (\Delta S) into executable evaluations.

For (n) evaluations, the agent’s pass rate is:

[ \text{Pass Rate} = \frac{\text{Passed Evaluations}}{n} \times 100 ]

This helps developers determine whether a source change has caused a real change in their agent’s behavior.

How I built it

I built Stale AI as both a command line tool and a local web dashboard.

The backend uses Python and FastAPI. The command line tool and dashboard share the same pipeline so that developers receive consistent results through either interface.

The system includes:

  1. Document change analysis
  2. Repository scanning
  3. Regression evaluation generation
  4. Repeated evaluation trials
  5. Pass and fail reporting
  6. Source hashing and local state tracking
  7. Deterministic fallback behavior
  8. Optional AI assisted repair
  9. Git diff generation
  10. GitHub Actions workflow generation

For the main demonstration, I used a change in the OpenAI Python SDK.

The older documentation instructed developers to use:

import openai

response = openai.ChatCompletion.create(
    model="gpt-3.5-turbo",
    messages=[
        {"role": "user", "content": "Hello"}
    ],
)

The updated documentation uses a client based interface:

from openai import OpenAI

client = OpenAI()

response = client.chat.completions.create(
    model="gpt-3.5-turbo",
    messages=[
        {"role": "user", "content": "Hello"}
    ],
)

The sample agent intentionally contains the outdated implementation.

Stale AI detects the documentation change, locates the stale code, generates three evaluations, and runs them against the agent.

Before repair:

[ \text{Pass Rate} = \frac{0}{3}

\times 100

0% ]

After repair:

[ \text{Pass Rate} = \frac{3}{3}

\times 100

100% ]

This creates a clear demonstration of the agent failing before the repair and passing after it.

Challenges I faced

Defining the right problem

One of the biggest challenges was deciding what Stale AI should and should not claim to solve.

Stale AI is not a universal hallucination detector. It cannot identify every reasoning mistake or unknown factual error.

For example, it would not automatically discover the PMOS name change unless the relevant medical source was being tracked.

Its purpose is more specific.

Stale AI detects when a trusted source changes and tests whether an agent still reflects the older information.

It works best for systems that depend on identifiable sources that change over time.

Making the demonstration reliable

Live AI requests can fail because of model settings, API configuration, rate limits, or unavailable external tools.

I added a deterministic fallback path so that the central workflow remains reproducible during a demonstration.

The live AI path remains available, but the entire project does not fail when an external service is unavailable.

Displaying the repair

The bundled sample repository was stored inside the main Stale AI repository.

Since the sample folder did not have its own .git directory, the repair worked, but the proposed Git diff initially appeared empty.

I updated the Git diff logic so Git can recognize the enclosing repository while limiting the displayed changes to the sample folder.

Reducing noisy scan results

A simple keyword scan can find both outdated terms and their correct replacements.

For example, a repository may contain:

openai.ChatCompletion.create

and:

client.chat.completions.create

Only the first reference is outdated.

This taught me that scan results must clearly separate:

  1. Outdated references
  2. Current replacements
  3. Unrelated matches

What I learned

The most important lesson was that AI reliability is not only about whether a model is intelligent.

An agent can be correct when it is released and become incorrect later, even when its own code has not changed.

The PMOS example made this problem real to me. The AI had not suddenly become less capable. The information surrounding it had changed, while its answer remained frozen in the past.

I also learned that detecting a changed document is not enough.

Developers need evidence that the source change affects the actual behavior of their application.

By converting source changes into executable evaluations, Stale AI connects changing information to measurable failures.

I also learned the importance of narrowing a product’s claim.

Stale AI does not solve every AI mistake. It solves one specific problem:

Detecting when a previously correct AI agent becomes outdated because its trusted sources changed.

What is next

The next step is to make Stale AI easier to use in real development workflows.

Planned improvements include:

  1. Accepting GitHub repository links directly
  2. Automatically identifying important source dependencies
  3. Monitoring documentation links for changes
  4. Improving semantic repository scanning
  5. Reducing false positive results
  6. Connecting every failed evaluation to the exact source change
  7. Opening pull requests with verified repairs
  8. Supporting internal company knowledge bases
  9. Running scheduled checks through continuous integration

The long term goal is to help developers treat AI knowledge like a software dependency.

It should be monitored, tested, and updated before people are forced to discover that it has become stale.

Built With

Share this project:

Updates