Inspiration
AI systems depend not only on code, but also on policies, APIs, documentation, and other external knowledge.
Traditional CI runs when code changes. But an AI output can become unsafe even when the code stays exactly the same — simply because the knowledge it depended on changed.
That led us to build KnowledgeCI: CI for the knowledge your AI depends on.
What it does
KnowledgeCI follows a simple causal pipeline:
Trusted Evidence → Knowledge Change → Impact Analysis → Isolated Replay → Safe Action
When trusted knowledge changes, KnowledgeCI identifies which AI outputs are affected, replays only those outputs against the new evidence, and proposes a bounded response:
- KEEP
- REVIEW
- REGENERATE
- PAUSE
High-risk actions remain proposal-only.
Sponsor integrations
Nutrient DWS
We use Nutrient Document Web Services to extract typed policy facts from documents.
In our demo, a refund policy changes its auto-approval threshold from $500 to $200. KnowledgeCI preserves historical correctness while flagging new or pending $420 decisions for review or pause.
SerpApi
We use SerpApi for discovery of updated external information.
For our Slack API scenario:
SerpApi = discovery Slack first-party evidence = authority
KnowledgeCI only accepts evidence after authority checks succeed.
Evidence-bound replay
Replay must use the exact evidence that triggered the change.
KnowledgeCI verifies evidence identity, revision, hashes, and canonical ordering. Missing, reordered, duplicated, conflicting, or incomplete evidence fails closed.
If we cannot prove what evidence the replay used, we do not trust the replay.
Selective regression
Instead of replaying every AI output, KnowledgeCI uses dependency lineage to replay only affected outputs.
In our fixture evaluation:
72 full replays → 32 graph-scoped replays → 16 selective replays
while preserving the same internal correctness result.
How we built it
Core components include:
- versioned knowledge and provenance
- output-to-evidence dependency lineage
- knowledge change detection
- impact analysis
- isolated replay
- evidence binding
- bounded remediation
- execution-backed evaluation UI
Challenges
The hardest problems were:
- separating discovery from authority
- preserving historical correctness after policy changes
- proving replay used the correct evidence
- failing closed on incomplete evidence
- avoiding unsafe autonomous remediation
- explaining the full causal chain clearly in a 3-minute demo
What we learned
AI reliability is not only a model problem.
An AI system can behave exactly as designed and still become unsafe because the world it depends on changed.
Provenance becomes much more useful when it drives impact analysis, selective replay, and bounded remediation.
What's next
Next steps include:
- more trusted knowledge connectors
- event-driven change ingestion
- richer agent dependency graphs
- review workflows for remediation proposals
- replay history and drift analysis
- CI/CD and observability integrations
KnowledgeCI — CI for the knowledge your AI depends on.
Log in or sign up for Devpost to join the conversation.