Relevance Incident Lab

Investigate how search results change across retrieval, deterministic ranking, and LLM reranking before deploying to prod

Comment

TL;DR

  • We see that LLM can rerank search results, and we should have a way of inspecting the whole chain from retrieval to results.
  • With DuckDB, Python with FastAPI, and GPT-5.6 (Luna), we performed search on synthetic pet food product data with ground truth labeling and item of interest highlighted as protected and inspect each layer in the Relevance Incident Lab (RIL)
  • We found that LLM reranking can be beneficial in some cases, and not others (i.e., hurting the ranking of protected items, or unchanged so that the LLM calls are additional costs we do not need)
  • Codex and GPT-5.6 were used extensively in planning and implementation of the code and synthesizing data, as well as the video demo via plugins and debugging the UI with skills
  • Future direction includes extending RIL's capabilities to be able to run as a containerized companion to a developer’s search stack, support hybrid, semantic, and AI-search pipelines, expose its evidence through an MCP server, and evaluate real public datasets beyond the current synthetic examples

Inspiration

Research such as RankGPT and RankZephyr showed that large language models can rerank search results. That made me wonder: when a ranking looks wrong, is it the LLM's fault or was the issue in retrieval or the deterministic ranking baseline, or some other parts such as output validation?

With that, we built Relevance Incident Lab (RIL) to make that whole chain inspectable end-to-end, to see between the layers what each is doing, an essential tool when we might be accusing LLM being the blackbox.

What it does

With DuckDB full-text retrieval, a deterministic baseline, GPT reranking, response validation, and other features like p95 latency, RIL lets a developer be able to trace the search result through each layer from end-to-end.

Each test case includes frozen relevance judgments (i.e., our version of ground truth) so RIL can show exactly which items moved, why a protected relevant item matters, and whether the change helped or hurt. The caveat regarding the suggestion and summary or final outcome is that they are pieces of evidence for a human reviewer, not an automatic release decision.

How we built it

We used DuckDB Full-Text Search and BM25 for lexical retrieval, Python with FastAPI and Pydantic for the evidence and validation layer, GPT-5.6 Luna through the OpenAI API for reranking, and React with Vite for the review interface. In addition to the rerunnable nature of the data, the package runs in Docker without an API key or network dependency, while the public site is a read-only demonstration of the same evidence.

Codex was part of the development workflow too. Sol at ultra reasoning helped refine the PRD and technical design, Goal mode helped with longer implementation and verification work, installed UI skills (i.e., Impeccable) helped polish the interface and accessibility, and HyperFrames helped produce the demo video. Moreover, we used Sites to deploy a static demo with ease.

Challenges we ran into

The hardest part was producing a truthful BLOCK case. Our first organic case did not regress. A later chicken-free case failed before reranking because its protected item landed at retrieval rank 14, outside the top-12 candidate pool. That was useful evidence: a reranker cannot rescue an item it never receives.

The puppy case exposed a different problem when early model attempts returned invalid or hallucinated product IDs. We redesigned the protocol around opaque candidate handles, exact validation, one bounded repair attempt, explicit abstention, and deterministic fallback.

Only after freezing a separately motivated beef-exclusion fixture (yet another case) before observing the provider result did we obtain a defensible BLOCK demonstration. This took longer than simply generating a convenient output, but it kept the evidence honest.

Accomplishments that we're proud of

We produced both sides of the story:

  • A GPT_PREFERRED case where the protected puppy result moved from baseline rank 11 to GPT rank 1.
  • A BLOCK case where the protected beef-exclusion result began at baseline rank 1 and the GPT reranker demoted it, triggering investigation.
  • A 100-pair response-time campaign for the qualified puppy case, giving us a real p95 measurement instead of relying on one fast call.
  • A provider-free recorded demo that preserves attempts, validation, replay, metrics, policy reasoning, and exact artifacts for review.

We are also proud that RIL keeps INCONCLUSIVE as a real outcome. Missing or invalid evidence does not silently become a success.

What we learned

LLM reranking can help, but it is only one part of the search pipeline.

  • Retrieval recall comes first: if the relevant item is outside the candidate pool, reranking cannot recover it.
  • An aggregate metric can improve while a business-critical protected item gets worse, so both views need to be evaluated.
  • Model output should be treated as untrusted input: IDs must be validated, and the system needs abstention and deterministic fallback behavior.
  • One successful response is not enough to establish latency or reliability; repeated, comparable runs matter.
  • Synthetic fixtures are useful for controlled regression tests, but they are not proof of production performance on real customer data.
  • BLOCK should remain an advisory, human-gated decision. RIL explains the evidence; a person owns the release action.

What's next for Relevance Incident Lab

  • Turn RIL into a containerized companion for a developer's existing search stack, with a documented fixture/import contract for bringing their own queries, candidates, judgments, and rankings.
  • Extend RIL beyond lexical retrieval to inspect hybrid, semantic, and broader AI-search pipelines.
  • Create an MCP server so a developer can connect a self-hosted RIL instance to Codex and ask questions about recorded incidents and evaluation results.
  • Evaluate broader public datasets such as the Amazon Shopping Queries ESCI dataset, then extend beyond product search to news and other changing corpora.
  • Add connectors that snapshot evidence from production search systems and run RIL as a human-reviewed search-quality gate in CI/CD, without giving it automatic release authority.

Built With

Share this project:

Updates