• 

  Preview

• 

  Mobile preview

Inspiration

AI assistants are great at producing answers, but not always at admitting when the evidence is weak or missing.

When someone asks a scientific question, a typical assistant may blend together academic research, blogs, news articles, forum posts, and its own background knowledge. The result can sound confident even when the underlying evidence is unclear.

I wanted to build the opposite: a system that treats uncertainty as a useful result.

That became ChatLibris.

No paper, no answer.

ChatLibris searches academic literature, synthesizes only the sources it retrieves, and returns one of three outcomes:

• Evidence supports this
• Evidence is mixed
• Answer unknown

“Unknown” does not mean a claim is false. It means the retrieved literature was not sufficient to establish an answer.

What it does

A user asks a question such as:

Does creatine supplementation improve muscular strength in healthy adults?

ChatLibris searches an academic paper index and retrieves relevant titles, abstracts, authors, publication years, and source metadata.

It then sends only that academic evidence to the language model.

The model does not receive Reddit posts, blogs, news articles, or general web-search results.

ChatLibris returns:

• A plain-language answer
• An evidence status
• Key findings
• Important limitations
• Citations linked to the retrieved papers

Conceptually, the verdict is:

[ \operatorname{verdict}(q,E)= \begin{cases} \text{Unknown}, & \text{if no retrieved papers directly address the question}\ \text{Mixed}, & \text{if relevant papers materially disagree}\ \text{Supported}, & \text{if the retrieved evidence supports a cited answer} \end{cases} ]

How we built it

ChatLibris is a full-stack Next.js and TypeScript application deployed on Vercel.

The pipeline is:

User question
↓
Academic literature search
↓
Filter and deduplicate papers
↓
Create an evidence packet
↓
Generate a structured synthesis
↓
Validate citations
↓
Return Supported, Mixed, or Unknown

We used the Semantic Scholar Academic Graph API to retrieve academic papers and abstracts.

Each retrieved paper is assigned a stable identifier such as P1, P2, or P3. The model receives those identifiers alongside the paper metadata and abstracts.

For synthesis, we used the OpenAI Responses API with structured outputs. The model returns a predictable object containing:

{
  "status": "supported",
  "confidence": "medium",
  "answer": "The retrieved evidence generally indicates...",
  "claims": [
    {
      "text": "A specific evidence-backed finding.",
      "sourceIds": ["P1", "P3"]
    }
  ],
  "limitations": [
    "Most studies involved younger adults."
  ]
}

The server validates every citation before displaying it. If the model references a source that was not actually retrieved, that citation is rejected.

ChatLibris also includes a deterministic abstention layer. If no usable academic evidence is found, the system returns Answer Unknown instead of asking the model to improvise.

Challenges we faced

Teaching the model to abstain

The hardest part was not generating an answer. It was preventing the system from answering when the evidence was insufficient.

Language models are optimized to be helpful, so they often fill in missing information. We had to make “Unknown” an explicit and valid product outcome.

We also added server-side rules so abstention would not depend only on prompt wording.

Preventing fabricated citations

A model can produce citation labels that look legitimate even when they were never included in the evidence.

To solve this, the server creates the source identifiers itself and checks every identifier returned by the model.

Claims without valid sources are removed.

Separating uncertainty from technical failure

An API timeout is not the same thing as an unknown scientific answer.

ChatLibris distinguishes between:

• No sufficient literature retrieved
• Mixed or inconclusive evidence
• Academic search failure
• OpenAI synthesis failure
• Missing deployment configuration

This prevents a technical error from being presented as a scientific conclusion.

Defining “academic”

Academic indexes can contain journal articles, conference papers, reviews, and preprints.

Because of that, we describe ChatLibris as grounded in indexed academic literature rather than claiming that every source is peer reviewed.

Building under time pressure

For the hackathon, we aggressively cut scope.

We did not build:

• User accounts
• A database
• Conversation history
• PDF uploads
• A vector database
• Full-text paper parsing
• A research agent

We focused on one complete experience:

Ask a question, retrieve academic evidence, and receive either a cited answer or an honest Unknown.

What we learned

The biggest lesson was that trustworthy AI cannot be created by adding a disclaimer after generation.

Trust has to be built into the architecture.

For ChatLibris, that meant:

• Restricting retrieval to academic sources
• Sending the model a bounded evidence packet
• Enforcing structured output
• Validating citations
• Supporting abstention
• Separating scientific uncertainty from technical failure

We also learned that uncertainty can be a product feature.

Most assistants compete on always having an answer. ChatLibris is designed to know when it should not answer.

What’s next

Future versions could include:

• Stronger peer-review filtering
• Multiple academic indexes
• Systematic-review prioritization
• Study-design classification
• Full-text analysis where licensing permits it
• Better comparison of conflicting papers
• Evidence quality scoring

The long-term goal is not to replace researchers or experts.

It is to make academic evidence easier to access without stripping away the uncertainty, limitations, and disagreement that are central to science.

Built With

• 5.6
• codex
• vercel