Why We Built EvidenceWiki

A note on “we”: Throughout this text, “we” refers to a human–AI collaboration. I set the direction, made the final decisions, and remained responsible for the outcome, while LLMs produced a substantial share of the implementation, testing, analysis, debugging, and iteration.

What Inspired Me

At the beginning of 2025, I began exploring the potential of LLMs in software development. Even then, the idea of an orchestrator that could independently handle complex tasks seemed especially promising.

The first tools were not particularly convenient for everyday use. However, the emergence of agentic development environments such as Cursor and GitHub Copilot opened new opportunities for automating software work. For me, the release of GPT-5.3 marked an important step forward: debugging, agentic and code-generation quality improved significantly, and models became more capable collaborators in the development process.

A post by Andrej Karpathy about building a local wiki for LLMs inspired me to develop agent-wiki-cli / python-wiki-llm. Its purpose is to turn a codebase into structured context that agents can understand and use efficiently.

This was my first project in which nearly the entire development lifecycle—from implementation to testing—was carried out with active LLM involvement. The project is now evolving toward automatic software-system documentation, but even its first version demonstrated an important shift in how development can be done.

What I Learned

I learned that modern models can do more than generate code. When they work within a structured process, they can help validate their output, identify defects, propose fixes, and support repeated development cycles.

I also learned that reliable autonomy requires more than capable models. Agents need structured knowledge, explicit workflows, and clear validation rules. A strong answer should not depend only on what a model can generate; it should be supported by evidence that people and other agents can inspect.

How We Built EvidenceWiki

My experience with agent-wiki-cli led to a new idea: a local, general-purpose research agent that uses wiki pages and collected sources as a structured knowledge base.

EvidenceWiki is not limited to software repositories or technical questions. It can support research in any domain where answers need to be grounded in traceable evidence—for example, science, standards, policy, legal and regulatory guidance, product decisions, and software development.

Its goal is not merely to produce an answer. The system evaluates available evidence, records provenance, estimates confidence, and returns results in a structured, machine-readable format. When the available evidence is insufficient, it can explicitly block the question and describe what information is still needed.

This makes EvidenceWiki useful in broader automated workflows. A software-development agent can use it to investigate a codebase, while a research agent can use it to explore external sources, compare evidence, support decisions, and pass reliable findings to other agents or human reviewers.

While building the project, we aimed to automate not only research and answer generation, but also testing, quality assessment, workflow coordination, and the decision-making process that determines whether a result is ready to be trusted or released.

Challenges We Faced

The central challenge was not simply generating answers with an LLM. It was building a research system that could determine whether its evidence was sufficient, explain why an answer could be trusted, and refuse to guess when it could not support a conclusion.

• Making answers auditable. A confidence score alone is not enough. Every answer needs to be linked to normalized source records, provenance information, and verifiable supporting quotes.

• Handling incomplete and changing evidence. Some questions require current official guidance, multiple evidence types, or source-specific validation. When those requirements are not met, the system must return a structured blocked state and a clear request for missing evidence—not a weaker or fabricated answer.

• Coordinating autonomous agents safely. Multiple agents may work on a research workspace at the same time. We needed deterministic question lifecycles, workspace locking, stale-claim recovery, and explicit state transitions so that automation remains reliable and testable.

• Treating external content as untrusted data. Research sources can contain misleading instructions, unsafe links, malformed files, or incomplete provenance. The system therefore treats source material as evidence rather than instructions and applies fail-closed acquisition and validation rules.

Looking Ahead

EvidenceWiki demonstrates that evidence-based autonomous workflows are no longer only a future vision. With capable models, structured knowledge, and strong validation, they are becoming practical today.

The goal is not to replace human judgment blindly. It is to give people and agents a more reliable way to research, validate, document, and make decisions using evidence they can inspect and reuse.

Built With

• codex
• python
• vscode