Paladin

Paladin changes its mind, only when the evidence earns it.

Comment

About the project

Inspiration

Scientific knowledge is supposed to change when better evidence arrives, but deciding how much it should change is difficult. A system that trusts every new result will constantly flip-flop, while one that clings to its existing beliefs will ignore genuine discoveries.

When we read the CORTEX BioSciences GROUND TRUTH challenge, the most interesting part wasn’t building a knowledge graph. It was protecting that graph while new, potentially unreliable research streamed into it. Evidence could be weak, retracted, outside the graph’s domain, or even contain instructions attempting to manipulate the system.

We built Paladin, the graph’s defender, it considers new evidence, changes beliefs when the evidence deserves it, and refuses changes that aren’t properly supported.

What it does

Paladin processes scientific results one at a time. For each result, it reads the current belief graph, examines the reported event and its structured provenance, and returns a set of validated changes. Paladin can:

  • Adjust a claim’s confidence according to evidence quality.
  • Give strong, independently replicated evidence more weight than a single unconfirmed result.
  • Hold extraordinary claims as pending until more independent evidence arrives.
  • Clear matching pending claims when evidence is retracted or fails replication.
  • Add a mechanism-specific exception instead of deleting a belief that remains generally true.
  • Detect discoveries that fall outside the graph’s current axes or transition regimes.
  • Reject prompt injections, fabricated provenance claims, conflicting reports, and unauthorized graph operations.

Incoming research text can help Paladin identify what scientific event occurred, but it can’t choose a claim ID, confidence value, operation, or provenance weight. Those decisions come from the read-only graph and trusted structured metadata.

How we built it

Paladin is implemented as a self-contained Python 3.10+ evidence policy. The deterministic path uses only the Python standard library and the types provided by the challenge framework.

Each evidence item moves through a staged decision pipeline:

  1. Normalize the text, including suspicious Unicode and invisible characters.
  2. Reject control instructions, hypotheses, questions, malformed input, and conflicting events.
  3. Parse and validate the structured provenance.
  4. Match the scientific event to claims or declared gaps in the graph.
  5. Decide whether to revise, scope, hold, dismiss, or flag the evidence.
  6. Pass every proposed operation through an independent authorization monitor.

Evidence Quality and Confidence Updates

We calculate evidence quality from five structured properties:

  • independent groups
  • replications
  • method directness
  • effect strength
  • method reliability

The scores d, e, and m are percentages from 0 to 100. The counts g and r are non-negative integers.

Evidence quality

$$ q = \min\left(1,\max\left(0, 0.40s(g) + 0.15s(r) + 0.20\frac{d}{100} + 0.15\frac{e}{100} + 0.10\frac{m}{100}\right)\right) $$

The count function applies diminishing returns:

  • s(n) = 0 when n is 0 or 1
  • s(n) = 0.40 when n is 2
  • s(n) = 0.75 when n is 3
  • s(n) = 1.00 when n is 4 or more

Evidence movement

For a contradiction:

$$ M = \min\left(2.8,\max\left(0,5(q-0.45)\right)\right)\left(0.55 + 0.45\frac{e}{100}\right) $$

For a strong confirmation when the prior confidence is below 0.70:

$$ M = 0.65 + 0.75q $$

For an ordinary confirmation:

$$ M = 0.15 + 0.30q $$

Define the evidence direction as follows:

  • direction = -1 for a contradiction
  • direction = +1 for a confirmation

The signed movement is capped at 3.0 log-odds:

$$ \delta = \operatorname{direction}\cdot\min(3.0,M) $$

Credible evidence updates confidence in bounded log-odds space:

$$ p' = \sigma\left(\operatorname{logit}(p) + \delta\right) $$

where:

$$ \operatorname{logit}(p) = \ln\left(\frac{p}{1-p}\right) $$

and:

$$ \sigma(x) = \frac{1}{1+e^{-x}} $$

Here, p is the prior confidence, p' is the revised confidence, and \(\delta\) is the signed evidence movement.

Important boundary condition

The strong-confirmation rule changes abruptly at p = 0.70. For example, with the same evidence quality q, a prior of 0.699 uses the strong-confirmation formula, while a prior of 0.700 uses the ordinary-confirmation formula. This can cause an unintended jump in the update size.

To avoid that discontinuity, either define “strong confirmation” independently of p, or replace the hard threshold with a continuous transition rule.

This lets strong contradictions produce meaningful movement while confirmations of an already-confident belief create only a smaller nudge.

Thin extraordinary evidence follows a different path. Paladin stores it as a versioned pending record containing a semantic fingerprint, bounded provenance values, and a hash of the evidence origin. Matching evidence from distinct origins can accumulate, while duplicate sources can’t inflate the result. Retractions and failed replications remove only the exact pending family they invalidate.

Every final operation is checked by an authorization monitor. It verifies that claim IDs exist, confidence values are finite and in range, scope changes match structured provenance, pending records are valid, and out-of-distribution proposals were already declared outside the graph’s model. If any operation fails validation, the entire result collapses to no_op.

We also built an optional Gemini 3.1 Flash Lite canary for a narrow set of structurally uncertain inputs. The canary receives only the raw evidence body and never the graph, provenance, claim IDs, or mutation API. It can return only benign, injection, or abstain, supported by verbatim quotes that Paladin verifies against the original text. It can’t create an event or propose a graph update. If the API is unavailable, times out, or returns malformed output, Paladin keeps the deterministic rejection.

For MLH

We used the Gemini API as an optional, fail-closed semantic “canary” for ambiguous or malformed evidence text. The Python backend calls google-genai’s client.models.generate_content() with deterministic temperature (0), JSON-only output, bounded tokens, and a request timeout. Gemini receives only the raw evidence body and classifies it as benign, injection, or abstain; it never receives provenance, graph state, claim IDs, or mutation APIs. Its response is verified by checking that quoted evidence appears verbatim in the input, while the deterministic parser remains responsible for event extraction, provenance weighting, and all graph mutations. Missing API keys, SDK errors, invalid JSON, timeouts, or failed grounding checks fall back to the deterministic no_op behavior.

Challenges we ran into

Reading the event without trusting the text

This was the hardest balance. Paladin needs the text to understand what happened scientifically, but that same text is untrusted.

Small language details caused surprisingly large reasoning errors. Early versions missed state aliases such as “pluripotency,” allowed unrelated clauses to flip the direction of a contradiction, and sometimes treated a state mentioned only in an exclusion phrase as a transition endpoint. We added stricter clause splitting, polarity handling, state resolution, and ambiguity checks to close those gaps.

Distinguishing contradictions from model limitations

A surprising result isn’t automatically out of distribution. A potency reversal may directly contradict an existing belief, while a transition between unrelated mature cell types may require an entirely new regime.

We solved this by comparing event endpoints with the graph’s topology and declared domain boundaries. Paladin flags an item only when it describes an excluded axis or a transition regime the graph genuinely cannot represent.

Defending against manipulative input

Our adversarial cases included embedded claim IDs, fake replication counts inside the body, direct mutation instructions, persuasive language, invisible characters, and Unicode lookalikes.

We initially focused on blocking known patterns, but learned that detection alone wasn’t enough. We added defense in depth: Unicode normalization, strict evidence admission, a closed mutation vocabulary, provenance-only weighting, and a final authorization monitor that doesn’t trust the body at all.

Letting an LLM help without giving it authority

The optional canary was useful for uncertain text, but connecting a model to a belief-revision system introduced another trust problem. We restricted its inputs, removed mutation-related fields from its response schema, grounded its evidence quotes, and sent its verdict back through deterministic checks.

We also encountered practical API failures, including an outdated model ID returning 404 and request deadlines below the API’s accepted minimum returning 400. These failures reinforced an important design rule: Paladin must behave safely and predictably when the model is completely unavailable.

What we learned

We learned that trustworthy belief revision isn’t just about choosing a clever probability formula. The system also needs to know which information is trusted, what each input is allowed to influence, and how to fail safely when it is uncertain.

Structured provenance turned out to be more useful than confident- sounding language. We also found that weak evidence shouldn’t always be accepted or discarded and holding it as pending preserves uncertainty without prematurely rewriting the graph.

Out-of-distribution detection taught us another form of scientific honesty: sometimes the correct response isn’t “true” or “false,” but “our current model doesn’t have the vocabulary for this yet.”

Most importantly, we learned to treat AI as an advisor rather than an authority. The optional model can help interpret an uncertain document, but deterministic policy and validated operations remain responsible for every change.

Our current offline test suite passes 33 automated tests, including belief revision, pending-evidence accumulation, exact retraction handling, OOD classification, canary routing, prompt- injection defense, and final authorization. The public practice scorer also passes the firewall gate and its OOD checks.

What’s next

Next, we want to:

  • Calibrate Paladin’s evidence weights against larger labeled research streams.
  • Expand its event parser to support broader scientific domains and state ontologies.
  • Grow the adversarial corpus with more semantic, multilingual, and Unicode-based attacks.
  • Add better shadow-mode evaluation for the optional canary so we can measure when it helps, abstains, or disagrees without allowing it to affect the graph.

Built With

Share this project:

Updates