Health Docs MCP Server

An MCP server that lets Claude Desktop answer questions about public health documents, with cited sources and a reliability indicator. Not medical advice.

Comment

Inspiration

I wanted an assistant that answers from documents instead of from memory, and says "I don't know" when the documents do not contain the answer. In health, an invented answer is costly, so every answer shows its sources. This is not medical advice.

What it does

The server exposes two tools to Claude Desktop through the Model Context Protocol (MCP):

  • search_documents: returns the 3 most relevant excerpts, with their source file and similarity score.
  • ask_question: classifies the question, writes an answer using only the retrieved excerpts, cites the source files, and returns a status: RELIABLE, TO VERIFY or NO ANSWER.

An intent router decides how each question is handled (definition, exercise, summary, comparison or off-topic). Off-topic questions are refused without searching or generating anything.

How I built it

  • Extraction: PyMuPDF, then 200-word chunks with a 40-word overlap.
  • Retrieval: local embeddings (multilingual-e5-small) stored in ChromaDB, cosine similarity.
  • Routing and answers: Mistral API (ministral-14b-2512), temperature 0, JSON output for the router.
  • Engineering: pytest tests, GitHub Actions CI, logs kept out of the MCP channel, blocking calls run outside the asyncio event loop.

Evaluation

I measured each change on 23 hand-annotated questions: 15 answerable, 5 on-topic but absent from the documents, 3 off-topic. An LLM judge from a different model family grades the answers.

  • Correctness: 73% -> 80% (11/15 -> 12/15)
  • Faithfulness to the excerpts: 77% -> 100%
  • Correct refusals on unanswerable questions: 8/8

Limits: the test set is small (one question is worth 6.7 points), and the reliability indicator is based on similarity scores, so it does not yet separate good answers from bad ones. These figures were measured on the original corpus of university course PDFs. The demo corpus is being switched to public-domain health documents, and the evaluation will be re-run on it.

Challenges

A local 7B model took about 2 minutes per answer and invented some. Switching to an API model fixed both. A different PDF library also removed glued words in the extracted text.

What's next

Health documents with verified licenses, a Docker image and a web demo.

Code: https://github.com/faraaawo-debug

Built With

  • chromadb
  • claude-desktop
  • github-actions
  • mcp
  • mistral-api
  • multilingual-e5-small
  • pymupdf
  • pytest
  • python
Share this project:

Updates