ResearchX

ResearchX helps researchers investigate scientific literature, verify claims against source evidence, uncover why experiments disagree, and identify what to explore next.

Comment

ResearchX: Building a Research System That Can Show Its Evidence

Inspiration

A research paper can report a result, but understanding how that result compares with other published work takes considerably more effort. Two papers may evaluate the same model family yet use different datasets, training configurations, splits, or metric definitions. A conventional search engine helps find the papers, and an LLM can summarize them, but neither step alone establishes whether a generated claim is actually supported by the source.

I wanted to explore a different approach: a research assistant in which retrieval, evidence validation, and uncertainty handling are part of the system's architecture rather than instructions left entirely to a language model.

That became ResearchX.

The Technical Approach

ResearchX is built around a hybrid retrieval and evidence-processing pipeline.

1. Hybrid retrieval

The retrieval system combines BM25 lexical search with dense vector search over paper passages. BM25 helps retrieve passages containing specific terminology, model names, and parameter values, while dense retrieval helps find semantically related material even when the wording differs.

ResearchX generates embeddings using Qwen3-Embedding-8B and stores 256-dimensional vectors in PostgreSQL with pgvector. The two retrieval rankings are combined using Reciprocal Rank Fusion, after which an NVIDIA Nemotron model reranks the leading candidates.

This combination was designed to handle both precise experimental questions and broader research queries. If embedding retrieval fails, the pipeline can continue in a degraded BM25-only mode.

2. Evidence validation outside the language model

The central engineering decision was to separate language generation from the rules that determine whether evidence is acceptable.

ResearchX uses NVIDIA Nemotron models for reranking, synthesis, and selected extraction tasks through Nebius Token Factory. The generated structured output is then checked by application code.

For a claim to survive citation validation, its quoted passage must appear verbatim in the cited source and contain the claim's relevant numerical information. Field-level extraction applies additional checks to values and experimental conditions, including datasets, splits, evaluation protocols, metric definitions, and training stages.

A model's confidence or fluent explanation is not sufficient to make a claim valid. If a claim fails validation, it is removed rather than treated as established evidence.

3. Contradictions and abstention

ResearchX does not assume that every question has a definitive answer in the available corpus.

The system tracks conflicting evidence and can return a CONFLICTING outcome when sources disagree. When the available evidence is insufficient, it can return ABSTAINED instead of filling missing information with a plausible value.

External scholarly discovery is triggered when the internal evidence is insufficient. Retrieved external material passes source checks before it can be used, and a search result does not automatically become a citation or a permanent corpus record.

This distinction between discovery, evidence, and validated claims is important: finding something relevant is not the same as proving a statement.

4. Research-specific analysis

Rather than treating every research task as a variation of chat, ResearchX provides eight specialized analyses:

  • Paper-to-Paper Diff: Compares papers field by field and distinguishes aligned experiments from materially different or insufficiently comparable ones.
  • Result Discrepancy Investigator: Examines differences between reported results and separates supported explanations from missing information.
  • Reproducibility Recipe Builder: Organizes the settings needed to reproduce an experiment while leaving unreported values unknown.
  • Next Experiment Planner: Helps identify follow-up experiments that could reduce uncertainty.
  • Claim-to-Evidence Auditor: Checks whether research records support a stated claim, value, or comparison.
  • Generalization and Transferability Analyzer: Examines how results and conditions change across evaluation settings.
  • Literature Review Completeness Auditor: Identifies areas that a research question or draft review may not cover.
  • Benchmark Reality Explorer: Classifies reported benchmark results according to their comparability.

These analyses share the same underlying evidence records and comparison logic rather than relying on eight completely independent chat prompts.

5. Traceability and architecture

The web application uses Next.js, React, and TypeScript. The Python FastAPI backend orchestrates retrieval, model calls, evidence checks, and analysis. Supabase provides PostgreSQL, pgvector, authentication, and storage.

Research Trace records the stages of a research run, including retrieval results, routing decisions, model attempts, validation outcomes, and fallbacks. Knowledge Map provides a way to explore relationships between papers, experiments, and research entities, with provenance attached to graph elements.

The model gateway routes tasks through Nebius Token Factory. Nemotron 3 Nano handles routine synthesis and reranking, while Nemotron 3 Super is used for comparison questions and selected full-text tasks. The routing policy is implemented in code, and outputs are validated before they enter the evidence pipeline.

Challenges

Balancing retrieval quality with practical constraints

Hybrid retrieval introduces more moving parts than a single search method. Lexical and dense retrieval have different failure modes, and their results must be combined and reranked without losing relevant evidence. The production embedding dimension was also chosen with the database's storage constraints in mind.

Extracting experimental facts without losing context

A value such as a learning rate or benchmark score is not meaningful in isolation. It belongs to a particular model, dataset, training stage, split, and evaluation protocol. Comparing values without that context can create false agreements or apparent contradictions.

ResearchX therefore represents experimental settings separately and validates extracted values against source passages and normalized conditions. When the conditions cannot be established, the system should preserve the uncertainty rather than infer a match.

Making model output dependable

Structured output alone does not guarantee factual correctness. A model can return valid JSON with an unsupported claim inside it. ResearchX therefore applies schema validation and separate evidence checks, with deterministic fallbacks when a model route fails or an output does not pass validation.

Integrating the full system

The application combines a web interface, API, database, retrieval indexes, model gateway, scholarly providers, and streaming traces. Deploying these components within free-tier infrastructure adds constraints around cold starts, service limits, and database capacity. The public deployment also has features that are deliberately disabled under current infrastructure limits, so the implementation and live configuration are not always identical.

Testing Credtionals

A shared account is available for judges and reviewers:

E-mail: [email protected]

Password: Judge-y-eIoSavnrZp-2026

What We Learned

The project reinforced that a useful research system is not just a language model connected to a collection of PDFs.

Retrieval determines which evidence the system can see. Data representation determines whether experimental conditions can be compared correctly. Validation determines whether a generated claim survives scrutiny. Abstention determines whether the system can acknowledge when the evidence is not enough.

We also learned that these layers need to be evaluated separately. Citation validity does not establish that every scientific conclusion is correct, and a technically complete pipeline does not automatically demonstrate superiority over existing research tools. ResearchX has evaluation results on small, author-built datasets, but broader validation against human-reviewed examples remains important.

What's Next

The next step is to expand human-reviewed evaluation sets for retrieval, numerical extraction, contradiction handling, and citation support. This would make it possible to measure not only whether a quote exists in a source, but also whether the resulting interpretation is correct and whether the system abstains appropriately.

The longer-term goal is to make the transition from literature discovery to evidence-backed research decisions more systematic.

ResearchX is built on one principle: a research answer should be inspectable, its claims should be traceable to evidence, and missing information should remain visible.

Built With

Share this project:

Updates

Submission history