FactKaji: An Incremental Knowledge Compiler for AI Agents

FactKaji is an incremental Knowledge Compiler that transforms evolving documents into trusted, traceable knowledge for AI agents.

Comment

FactKaji: An Incremental Knowledge Compiler for AI Agents

Inspiration

Large Language Models are improving rapidly, but the knowledge they consume is still managed in a surprisingly primitive way.

Most AI systems use Retrieval-Augmented Generation, or RAG: search a collection of documents, retrieve the most relevant passages, and place them into the model’s context window.

This works well when documentation is static. Real projects are not.

Knowledge changes continuously:

  • a new decision replaces an earlier one;
  • an experiment contradicts a previous result;
  • a correction updates only part of an old document;
  • the same fact appears across Markdown notes, ADRs, tickets, wikis, and source control;
  • outdated and current statements remain equally searchable.

Traditional retrieval can find both versions, but it does not know which one should be treated as current.

An AI agent may therefore receive two contradictory passages and confidently repeat the obsolete one.

We started with a different question:

What if knowledge were compiled like software instead of retrieved like documents?

That idea became FactKaji.

What it does

FactKaji is an incremental knowledge compiler for AI agents.

Instead of exposing raw documents directly to a model, FactKaji transforms evolving Markdown knowledge into smaller, state-aware, provenance-backed Knowledge Views.

Markdown and reference artifacts
              ↓
       Event extraction
              ↓
   State reduction and conflict detection
              ↓
    Task-specific Knowledge View
              ↓
           AI agent

FactKaji tracks how knowledge changes over time.

It can distinguish between:

  • decisions;
  • observations;
  • corrections;
  • rejections;
  • historical values;
  • active values;
  • disputed values;
  • retired knowledge.

When an old document says the primary datastore is MySQL and a later decision changes it to Postgres, FactKaji does not simply retrieve both passages. It compiles the current state and returns Postgres with its provenance, while preserving the historical record.

Core capabilities include:

  • incremental compilation of changed Markdown files;
  • provenance for every compiled fact;
  • supersession and correction handling;
  • automatic dispute detection;
  • version-aware current-state reduction;
  • task- and project-scoped Knowledge Views;
  • cited reference-document retrieval;
  • MCP tools for AI agents;
  • a dashboard for inspecting and settling disputes;
  • deterministic rebuilds, even when an LLM is used during extraction.

The result is a smaller and more reliable context that tells an agent not only what was written, but what should currently be believed.

How we built it

We designed FactKaji as a compiler rather than a document search engine.

1. Artifact ingestion

FactKaji accepts evolving Markdown sources such as:

  • ADRs;
  • design notes;
  • technical documentation;
  • experiment reports;
  • operational runbooks;
  • agent-generated knowledge;
  • Obsidian-style vaults.

Whole reference documents can also be attached without converting every sentence into a fact. This creates two complementary retrieval lanes:

  • structured facts for state, supersession, and conflict;
  • cited passages for detailed reference material.

2. Event extraction

Documents are converted into structured knowledge events.

The deterministic extractor handles explicit statements quickly and reproducibly. A hybrid model lane can process natural prose that does not follow the structured grammar.

The shipped implementation supports:

  • OpenAI models, including GPT-5.6;
  • local OpenAI-compatible model endpoints;
  • deterministic-only operation;
  • hybrid deterministic and model-backed extraction.

In a 44-note realistic-prose benchmark, the deterministic parser achieved 0% recall on off-grammar facts, while the Codex-backed hybrid extractor achieved 54%.

3. State reduction

Events are reduced into canonical subject state.

The reducer determines whether a subject is:

  • active;
  • observed;
  • disputed;
  • unknown or retired.

It also preserves the source events that produced the state, so every result can be explained and traced back to Markdown.

Conflicting authoritative values become explicit disputes rather than silently overwriting one another.

4. Knowledge View compilation

FactKaji does not return the entire knowledge graph for every request.

It compiles a bounded Knowledge View for the current task, including:

  • relevant current facts;
  • unresolved disputes;
  • source identifiers;
  • provenance;
  • supporting reference passages;
  • coverage and truncation metadata.

This gives an agent a focused context rather than a pile of loosely related documents.

5. Incremental compilation

When a source file changes, FactKaji recompiles only the affected artifacts, events, and subject states.

A central invariant of the project is:

incremental compilation == full rebuild

For the same source tree and committed extraction cache, both paths must produce the same state version and compiled knowledge.

The model lane is made reproducible through a content-addressed extraction cache. A model may be nondeterministic when first called, but its accepted output is frozen at rest and replayed during subsequent rebuilds.

6. Agent and dashboard interfaces

FactKaji exposes its capabilities through:

  • a CLI;
  • an MCP server;
  • knowledge retrieval and explanation tools;
  • fact and reference write paths;
  • task checkpoints;
  • evaluation tools;
  • a self-contained knowledge-graph dashboard.

The live dashboard can display a disputed subject, show its competing values and sources, and settle it through the same production write path used by the MCP tools.

The settlement creates a correction in Markdown, recompiles the affected state, and updates the subject from disputed to active.

Challenges we ran into

The hardest problem was not retrieval. It was defining what “current knowledge” actually means.

A newer document does not always invalidate an older document.

Two values may differ because:

  • one is historical and one is current;
  • they apply to different projects;
  • they use different scopes;
  • they are observations rather than decisions;
  • both are valid under different conditions;
  • one is only a formatting variation;
  • the sources genuinely disagree.

We therefore had to build an explicit event and state model rather than rely on timestamps alone.

Another major challenge was preserving determinism while using an LLM.

A naïve model-backed extractor can produce one fact during incremental compilation and a different fact during a full rebuild. That would make the compiled state unreliable.

We solved this by placing all model extraction behind the same invariant-preserving hybrid path and committed content-addressed cache.

Adversarial review also exposed several issues that normal happy-path testing missed, including:

  • an extraction mode that bypassed the cache and broke rebuild equivalence;
  • provenance paths that could reintroduce stale values;
  • an unsafe dashboard settlement endpoint;
  • Japanese retrieval failures caused by tokenization assumptions;
  • incomplete Knowledge Views that could mislead a model by appearing exhaustive.

These findings changed both the implementation and the claims we make about the project.

Accomplishments that we're proud of

The accomplishment we are most proud of is that FactKaji is no longer only an architectural idea.

It is a working end-to-end system with:

  • incremental Markdown compilation;
  • current-state reduction;
  • explicit disputes;
  • source-level provenance;
  • fact and passage retrieval;
  • MCP integration;
  • OpenAI and local model support;
  • a live settlement dashboard;
  • reproducible full rebuilds;
  • real-world dogfooding and evaluation suites.

We also validated the project through several types of evidence.

Real-vault dogfooding

FactKaji was run against a real multi-note knowledge vault rather than only synthetic test fixtures.

The system found genuine disagreements across documents, exposed extraction and canonicalization failures, and produced fixes that were then added to the regression suite.

Hybrid extraction evaluation

On a realistic-prose benchmark, the model lane recovered facts that the deterministic grammar could not extract, while the committed cache preserved deterministic rebuild behavior.

Retrieval and knowledge-evolution benchmarks

We tested FactKaji against:

  • stale and superseded documentation;
  • conflicting values;
  • tight context budgets;
  • large histories with tens of thousands of extra notes;
  • Japanese-language retrieval;
  • natural Obsidian vaults;
  • official-domain holdout questions.

The results also revealed limitations. In one official holdout, the raw score increased, but the evidence was not strong enough to attribute the improvement to FactKaji. One incomplete Knowledge View contributed to a regression. We documented the failure and expanded the source corpus instead of presenting the score as a success.

Bidirectional adversarial review

Codex and Claude were used to attack each other’s implementations and claims.

Codex independently refuted an overstated benchmark conclusion, which we corrected.

Claude reproduced a Codex-originated provider design that bypassed the extraction cache and broke the incremental/full-rebuild invariant.

Confirmed fixes were returned for another cross-model review pass.

The goal was not model agreement. It was to make unsupported claims and fragile implementations fail before submission.

What we learned

The most important lesson was that retrieval is only one layer of the knowledge problem.

Finding a relevant sentence is not enough.

An AI system also needs to know:

  • whether the sentence is still current;
  • what replaced it;
  • which source established the change;
  • whether the value is disputed;
  • whether multiple values are valid under different scopes;
  • whether the available context is complete enough to support a conclusion.

We also learned that knowledge infrastructure for AI shares many properties with software build systems:

  • source artifacts;
  • parsing;
  • intermediate representations;
  • dependency-aware updates;
  • reduction;
  • deterministic builds;
  • versioned output;
  • provenance;
  • regression testing.

This suggests that reliable agent knowledge may require a discipline closer to compilation and continuous integration than conventional search.

We think of that emerging discipline as KnowledgeOps.

What's next for FactKaji

FactKaji currently proves the core architecture, but there is much more to build.

Next steps include:

  • clause-level hybrid extraction to avoid artifact-level fallback blind spots;
  • richer scope and temporal reasoning;
  • multi-repository knowledge compilation;
  • Git-native change and review workflows;
  • automatic alias and identity repair;
  • improved dispute-settlement semantics;
  • model-specific Knowledge View optimization;
  • stronger causal evaluations across real agent tasks;
  • hosted and team-oriented deployments;
  • an open plugin and adapter ecosystem.

Our long-term goal is to make FactKaji a dependable knowledge layer for coding agents, research agents, enterprise assistants, and long-running autonomous systems.

Future AI systems should not repeatedly rediscover the contents of their documents. They should compile the knowledge those documents represent.

Built With

Share this project:

Updates