Learning Buddy

Inspiration

I found MCP and LangGraph so cool so wanted to build something with it that could actually help me and every student, in whatever they are trying to learn.

What it does

Takes your pdfs, converts them into embeddings with appropriate metadata. Being exposed as an MCP server it attaches very simply to Claude, helping you recall, pin-point or test you on things you have embedded.

How I built it

Using LangGraph, FastMCP, Langchain, Python and Pydantic as technologies, combined with my knowledge of RAG and Agentic AI and Problem solving in general, I divided the problem into 3 sections:

Section 1: Embeddings and Metadata: I initially analyzed the pdf structures of research papers and my university professors and created a function to extract text, figure out colors and fonts and map the right embedding to exactly the right chunk. I later created a revised (this) version to be more general and take any pdf and convert it into embeddings and chose lighter retrieval and embeddings models so almost any hardware can run it without sacrificing much performance.

Section 2: Retrieval Pipeline: I used ChromaDB as a persistent local vector database and cosine similarity to retrieve the most relevant PDF pages for a user’s query. The system initially retrieves up to 25 candidates, filters out weak matches using a similarity threshold, and returns the most relevant results. When there are many suitable matches, it uses a lightweight cross-encoder reranker to select the best five passages. This keeps retrieval accurate while avoiding the cost of loading a larger model for every request.

Section 3: LangGraph and MCP: I used LangGraph to organise the query flow into clear steps: first understanding and structuring the user’s question, then retrieving relevant passages from the local vector database. Pydantic defines the structured query format and optional metadata filters such as PDF filename and page number, making the system more reliable and less likely to invent filters. Finally, I exposed the workflow through FastMCP as an MCP tool, so an AI assistant can search a user’s private local PDF collection through natural-language questions without uploading the documents or embeddings.

Challenges I ran into

Pdf structures: Almost every non standardized pdf has a different structure so even though my personal model is working very well for me and my entire cohort, using a rigid plan that works for my university format doesn't make sense, so I had to think of ways to generalize it. I went with simple embeddings to simplify the architecture but I also built a version that uses a LangGraph flow to appropriately assign metadata to the text, this version also vastly outperformed the current generic 1.0 version but given it had more latency and required strong hardware I went with this version that almost anyone can use.

MCP Problems: Claude had trouble detecting the server and even after tunneling wasn't able to run the models, even though on manual runs the architecture was fine, so I experimented with a lot of embedding and retrieval models to figure out the right balance for the weight vs performance tradeoff.

Accomplishments that I'm proud of

Built a complete local PDF-to-RAG pipeline, from extraction and embeddings to retrieval through an MCP tool. Made the system privacy-conscious: PDFs, embeddings, databases, and API keys remain local and are excluded from submissions. Improved the original project into a portable version that works with any PDF collection instead of one specific dataset. Balanced quality and accessibility by using lightweight embedding and reranking models that run on ordinary hardware. Tested the pipeline on 25 original PDFs, indexing 252 chunks and successfully returning relevant retrieval results.

Apart from this and the fact that all 100 of my cohort mates are now using it and I won a university prize for the model, I believe I made a huge leap towards a more general architecture that can use AI to help people to learn and not take their cognitive load, further worsening their memory, recall and reasoning. Next steps include making a fast and light enough LangGraph flow to better embed metadata into any text based on context, previous pages and structure and use traditional search algos like BM25 to truly make this a triple threat. Semantic retrievals: RAG, Query retrievals: Metadata based search, Frequency retrievals: BM25 and then all the results ranked by the re-ranker and combined with the intelligence of Claude to give a truly smart and context aware search experience to assist learning.

What I learned

That how the data is ingested matters more than how big your embedder or re-ranker is, the goal is stronger retrieval and novel methods while much stronger in many sense should not replace traditional methods in areas where the simple and traditional methods work well, rather, the 2 should be combined and it must be figured out how to make them each others' strength rather than a bottleneck. Because at the end of the day, it's the users that must get value from your product and not you showcasing complicated architectures just because you can, and that's where all the complexity lies.

Also that, while building an MCP you cannot go with the biggest models even if they work on your set-up, the goal is to make them available to everyone but still functional.

What's next for Learning Buddy

A truly general agentic embedding pipeline, which I have already tested and gives even better results by 17% and making privacy a bigger concern, adding authentication and authorization along with a frontend and keeping up with search engine related research papers and experiment on my own to iteratively improve it.

Built With

• agent
• agentic
• agenticai
• ai
• chroma-db
• fastmcp
• grok
• information-retrieval
• langgraph
• llm
• python
• rag