Inspiration
Large language models are extremely good at producing plausible explanations. That is useful when answering ordinary questions, but dangerous when investigating incidents, failures, or other high-stakes events.
Once a model finds a coherent explanation, it can reinforce that explanation instead of actively looking for evidence that might disprove it. This resembles confirmation bias in human investigations.
We built PRIOR around a different question:
What if an AI system had to earn confidence by trying to prove itself wrong?
PRIOR treats investigation as an evolving, falsification-driven process—not a one-shot answer. Evidence stays separate from interpretation, alternative hypotheses remain visible, uncertainty is explicit, and every conclusion retains its lineage.
What it does
PRIOR transforms a complex event into a structured causal investigation.
It produces:
- an executive causal assessment;
- a canonical evidence ledger;
- an expectation matrix;
- anomalies and missing observations;
- competing hypotheses with confidence scores;
- explicit refutation conditions;
- a hypothesis graveyard;
- the highest-value missing evidence;
- the next test with the greatest discriminating power;
- a versioned learning diff when evidence changes;
- complete audit and lineage metadata.
Evidence Scout extends this process through governed external search.
PRIOR first identifies a specific evidence gap. A human must explicitly authorize the search. The system then performs a bounded web search and returns a small set of candidates with source context and rationale.
Those candidates do not automatically become evidence. A human must accept or reject each one before it can enter a reinvestigation.
Our central product rule is:
Search never becomes evidence automatically.
How we built it
PRIOR is a Next.js and TypeScript application powered by OpenAI GPT-5.6 using strict Structured Outputs.
Every investigation must conform to a canonical JSON Schema and is validated with AJV before reaching the interface. The same structured snapshot powers three workspaces:
- Executive for decisions and causal assessment;
- Investigator for evidence, expectations, anomalies, hypotheses, and tests;
- Audit & lineage for provenance and reproducibility.
CockroachDB is the durable source of truth for investigation snapshots, Evidence Scout actions, candidates, and candidate-to-evidence links.
The governed Evidence Scout workflow uses:
- AWS SQS for asynchronous dispatch;
- AWS Lambda for bounded search execution;
- OpenAI
web_searchfor external evidence discovery; - AWS Secrets Manager for credentials;
- CloudWatch for sanitized structured logs;
- partial batch failures and a DLQ for recovery;
- AWS SAM and CloudFormation for infrastructure.
Cost and autonomy are bounded in code: at most two searches and five candidates per action, batch size one, daily action limits, and maximum Lambda concurrency three.
We built the project as horizontal end-to-end slices with Codex: contract, reasoning, UI, persistence, governed search, cloud execution, observability, and live verification.
Challenges we ran into
The hardest challenge was making every boundary real rather than simulated.
Our first Lambda deployment used an ESM bundle that Node interpreted incorrectly. Moving to .mjs fixed the initial loader error, but the pg dependency required dynamic CommonJS imports. We corrected the Lambda build boundary by producing a CommonJS bundle without changing the application’s global module configuration.
The next challenge was observability. The SQS handler correctly returned partial batch failures, but its catch path hid the underlying CockroachDB error. We added sanitized per-record structured logging without exposing payloads, URLs, credentials, or API keys.
That revealed a subtle database configuration issue: a host=base query parameter overrode the real CockroachDB hostname. The connection string also contained a local sslrootcert path that could not exist inside Lambda. Removing those overrides while preserving sslmode=verify-full restored secure connectivity.
We also found a concurrency race in local tests caused by a mutable global candidate store. We replaced the hidden dependency with explicit injection and ran the complete suite three times to prove the fix was stable under parallel execution.
Accomplishments that we're proud of
We are most proud that PRIOR is not just a UI prototype.
We verified the complete deployed round trip:
SQS → Lambda → OpenAI web_search → CockroachDB
The live action completed with one search call and four candidates, remaining inside every configured governance limit.
We also delivered:
- explicit human authorization before external action;
- human acceptance before search results become evidence;
- database-backed idempotency;
- bounded concurrency and cost;
- durable snapshots and lineage;
- sanitized production observability;
- retry and DLQ recovery;
- 40/40 passing test files;
- three consecutive full-suite passes;
- passing TypeScript, ESLint, production build, SAM build, and SAM validation.
What we learned
The largest lesson was that trustworthy AI is primarily a systems-design problem.
A better prompt is not enough. Trust emerges from explicit boundaries:
- what the model may infer;
- what counts as evidence;
- when external action is permitted;
- how cost is bounded;
- who accepts new information;
- how failures are recovered;
- how every conclusion can be traced.
We also learned to treat evaluation artifacts as evidence about the system itself. When an evaluation failed, the correct response was not automatically to change the prompt. In one case, the investigation was valid and the evaluator was measuring the wrong property.
Finally, we learned that human-in-the-loop should not be a decorative approval button. Human decisions must change what the system is technically allowed to do.
What's next for PRIOR
Next, we want to expand PRIOR from a powerful investigation console into a collaborative causal-intelligence platform.
Our roadmap includes:
- additional real-world domains beyond infrastructure incidents;
- collaborative investigation workspaces;
- role-based authorization and review;
- richer source provenance and evidence-quality scoring;
- visual hypothesis and evidence graphs;
- configurable investigation policies;
- organization-level budgets and governance;
- automated regression evaluations across case libraries;
- deployment-ready multi-tenant security;
- integrations with incident, intelligence, and compliance workflows.
The long-term goal is not to build an AI that sounds certain.
It is to build an investigation system that makes uncertainty useful—and confidence accountable.
Log in or sign up for Devpost to join the conversation.