Digilib-MCP

The MCP that enables your LLM to find and cite real sources for your research

Comment

Inspiration

DigiLIB started as a browser-based Flask chat app that queried a single source (OpenAlex) and generated a PDF. While functional, it had three major problems:

  • Interface lock-in: The literature search engine was trapped inside a single browser tab instead of being available across agentic toolchains and developer environments.
  • Coverage gaps: Relying on one source meant missing preprints, domain-specific indexing, and books.
  • Security liabilities: Connector code carried hardcoded credentials and lacked a decoupled transport layer.

We realized the real value was not another standalone chat UI, but the underlying capability: finding literature across diverse academic sources, deduplicating records, and turning them into publication-ready write-ups. We stripped away the frontend and rebuilt DigiLIB as a universal Model Context Protocol (MCP) server so any client—Claude Code, ChatGPT, Cursor, or autonomous agents—can use it directly.

What it does

digilib-mcp is an open-source MCP server that gives AI agents dedicated academic retrieval and synthesis capabilities. It provides three primary tools, along with MCP prompts and resources:

  • search_papers: Concurrently queries four scientific databases: OpenAlex, arXiv, Semantic Scholar, and Crossref.
  • search_books: Concurrently searches Open Library and Google Books for foundational texts.
  • export_research_tool: Compiles findings, summaries, and citations into structured, publication-grade LaTeX source code.
  • DigiLIB Researcher Persona: Delivered natively through MCP prompt primitives so host models can adopt an academic research persona out of the box.

Query Enhancement Pipeline

Instead of the main LLM manually writing queries for every individual source, it simply passes high-level keywords along with conversational context. A fast, lightweight secondary model (NVIDIA Nemotron 3.5 Lightning via OpenRouter) expands these into source-optimized academic and book search terms. If this enhancer model is unreachable, the system automatically falls back to the original keywords so the search never hard-fails.

Normalization and Deduplication

Incoming search results across all six providers are standardized into a uniform schema. Duplicate records are filtered out by title on a first-seen-wins basis, giving the agent a clean, unified list of references.

How we built it

digilib-mcp is built in Python using the official MCP SDK and designed for asynchronous throughput and zero local rendering dependencies.

  • Server Core: Built with Python MCP SDK (MCPServer) supporting both local standard I/O (stdio) and remote streamable HTTP transports.
  • Async Concurrency: Utilizes httpx and asyncio.gather to query REST APIs in parallel.
  • Preprint Adapter: Runs requests inside worker threads via asyncio.to_thread to handle blocking issues.
  • Query Expansion: Integrated NVIDIA Nemotron 3.5 Lightning via OpenRouter for intent-to-query routing.
  • LaTeX Generator: An in-memory string templating engine that returns raw .tex back to the host LLM without requiring local pdflatex or system TeX distributions.
  • Auth Gateway: Custom ASGI middleware running constant-time token comparison for secure remote deployment.

Challenges we ran into

  • arXiv WAF Fingerprinting (HTTP 406): arXiv's edge security flagged and blocked httpx async requests with 406 Not Acceptable. We resolved this by routing arXiv queries through standard synchronous requests inside asyncio.to_thread.
  • Reverse Proxy & Host Header Mismatches (HTTP 421): When deploying to Railway, the MCP SDK's transport security rejected requests with 421 Invalid Host header because it expected localhost. We resolved this by configuring TransportSecuritySettings(enable_dns_rebinding_protection=False) in gateway mode, relying on the reverse proxy's edge and our token middleware for security.
  • SDK Upgrades: Adapting to breaking changes in the fast-evolving MCP Python SDK (such as FastMCP transitioning to MCPServer).
  • Securing Remote Endpoints Without Full OAuth: We built a lightweight ASGI BearerTokenGate middleware using constant-time string comparison (hmac.compare_digest) to protect the server and OpenRouter budget from unauthorized access.

Accomplishments that we're proud of

  • Protocol-First Interoperability: Any MCP client can now perform full-stack academic discovery with zero custom client-side glue code.
  • Resilient Graceful Degradation: The pipeline avoids hard failures—if the Nemotron enhancer times out or an API endpoint throttles, the server still aggregates and normalizes available literature.
  • Zero-Dependency LaTeX Generation: Emitting raw LaTeX strings directly to the calling LLM keeps the server container lightweight and portable.
  • Production-Ready Gateway: Successfully configured and deployed a secure, edge-friendly remote MCP server over HTTP.

What we learned

  • Headless Architecture Over Web UIs: Protocol-based tools provide much wider utility than isolated web applications, making capabilities universally accessible across IDEs, CLIs, and web-based LLM clients.
  • Reverse Proxy Mechanics: Exposing services via reverse proxies requires explicitly separating host-header validation, transport-level protections (DNS rebinding), and application-level authentication.
  • Sub-Agent Specialization: Delegating query reformulation to a smaller, faster model improves search accuracy across heterogeneous search backends while keeping latency low.

What's next for Digilib-MCP

  • Full-Text Parsing: Adding tools to retrieve open-access PDFs and extract full methodology sections instead of abstracts alone.
  • Citation Graph Traversal: Enabling forward and backward citation network discovery via Semantic Scholar and OpenAlex APIs.
  • Reference Manager Sync: Direct export to Zotero libraries and .bib files.
  • Local Semantic Caching: Implementing local vector embeddings for fast multi-hop semantic querying across retrieved literature sets.

Built With

Share this project:

Updates