Inspiration
CORTEX's own framing hooked us: a research-intelligence platform where an LLM proposes but a knowledge graph decides, with a hard rule that no amount of persuasive text can ever rewrite what the system knows. That's not a toy problem, it's basically the trust question every AI-plus-data-source system has to answer eventually. We wanted to see how far a genuinely careful, rule-based decision policy could get before reaching for anything fancier.
What it does
Feed it a stream of new "results" about a cell-biology knowledge graph, one at a time. For each one it decides: does this confirm or contradict something the graph already believes, and if so by how much; is the evidence too thin to trust yet; is it describing something the graph was never built to represent; and — no matter what the text says or how it's phrased — could it be an attempt to just tell the system what to believe. It only ever acts through a small set of pre-approved, validated operations, never by editing anything directly.
How we built it
One Python function, ingest(), no ML model, no dependencies beyond the standard library. Confidence updates happen in log-odds space so evidence naturally adds up and can never overshoot to false certainty. How strong a result is comes entirely from structured metadata (how many labs, how replicated, how direct the method) — never from how convincing the write-up sounds. Whether something is a real contradiction versus "outside what this graph even models" gets decided from actual structural properties of the data, not keyword vibes. And the text itself is only ever allowed to say which claim is being discussed, never what number to set it to.
Challenges we ran into
The sneakiest bug wasn't in the logic, it was in the order evidence arrived. A skepticism check that worked fine in isolation started failing once an earlier item had already moved a claim's confidence, because the gate was checking the wrong signal. Fixing it meant realizing the threshold needed to key off something that doesn't drift, not the value it's supposed to be protecting. We also caught ourselves shipping a "fix" that was backwards (an AND where an OR was needed) and only found it by deliberately trying to break our own logic with an adversarial test case before trusting it.
Accomplishments that we're proud of
Every one of the four scored capabilities has a real, deliberate mechanism behind it, not just a rule that happens to pass the visible examples. We stress-tested with our own extra scenarios beyond what was given, found and fixed two real bugs that way, and every fix was verified end-to-end before moving on. The whole thing stays fully deterministic and dependency-free, and we wrote down, honestly, exactly where it's still guessing instead of overstating what it can do.
What we learned
That good calibration is mostly about discipline, not cleverness: refusing to look at the persuasive text, refusing to trust a single flashy result, refusing to let one good guess overwrite another. A lot of what would normally require a trained classifier turned out to have a cheap, honest, auditable substitute once we looked for the actual structural signal instead of a shortcut.
What's next for Patrick Molka Cortex ground truth solution
Swapping the current keyword-matching for real semantic retrieval so it can recognize a claim no matter how differently it's worded, widening and eventually validating the out-of-model vocabulary against real examples instead of guesses, and following the project's own intended production shape, an LLM doing careful extraction under a locked-down schema, feeding into the same auditable decision core, rather than either extreme of pure regex or an unconstrained model call.
Log in or sign up for Devpost to join the conversation.