MEMUX

An AI powered knowledge base platform for individuals & teams.

Comment

Inspiration

In the month of April 2026, Andrej Karpathy tweeted about how he uses Obsidian along with Claude Code to build his knowledge base. Reading this, I thought - what if we could have that combined in one single application? I already struggled with memory provision to my AI apps like chatgpt, claude, etc. these have a very loose memory layer, and are not suitable for retention of a complete knowledge base for future re-discovery of our own knowledge.

What it does

  • MEMUX is the complete cognitive layer for user, saving all the content dumped by user in a smart manner such that it is easy to navigate and retrieve.

  • It enables users to save hours of time they spend rediscovering their own knowledge. a image they save somewhere in a folder on their machine, or drive, or on whatsapp, or on slack, or on .... [list goes on]

  • With MEMUX, if you are re-discovering your own information - its just ONE prompt away. Looking for someting? a file ? A small piece of info in one of those images? a diagram that was there in a research paper? MEMUX handles them all. conduct a deep research on your knowledge base and reports in seconds.

  • Not only does it gives you raw data (citation & link to the original corpus referenced) but also widgets - like charts, timelines, etc.

How we built it

MEMUX have multiple 3 of operation:

  1. Indexing:

  • users can dump their important data directly in the app. AI agent indexes the corpus and saves to graph knowledge base.

  • MEMUX chrome extension is also there which auto-scrolls websites and captures screenshots (on user request) and sends it to indexer.

  • corpuses can be in "groups": if multiple pdfs, images, csv files, etc. are related, user can upload them together.

  • Indexer AI agent extracts text & structure from each image, if its pdf -> each page converts to image. if its website capture -> screenshots + DOM Tree.

  • Agent mentions its scratchpad where it makes all the notes regarding the corpus, one image / tree at a time. then the loop starts to build the knowledge base.

  • Agent writes multiple markdown files, section. by section, and a md file consisting a Table of Contents for that group / file / capture.

  • once we have multiple markdown files, agent link them, and hence start summarizing each of them. these are then vectorized and stored to the pglite pgvector vector store.

  • importand keywords are also extracted per corpus and vectorized for further reference.

  1. Knowledge Base

  • AI Agent indexes files provided by user using Gemma4 (Text + Vision + DOM Tree of websites) and builds knowledge base (Obsidian style linked markdowns)

  • User can also navigate, read, add, and modify content the knowledge base manually.

  • in addition to .md files, it also can store complete PDF, CSV, and image files. the files are stored in browser OPFS.

  • the knowledge base is also vectorized at time of indexing. summary of every md file is vectorized and stored in the browser pglite using pg-vetor.

  1. The Retriever Agent.
  • A chatgpt like window, but fully integrated with the internal tools.

  • users asks the agent about some information, the agent spins up: i) knowledge base exploration ii) vector search for related words / sentences iii) BM-25 search

  • it then goes through the results by sliding window approach (2-5 results in one window) and compiles the useful information + references.

  • Finally gives a grounded response with proper citations and widgets.

Challenges we ran into

  1. Designing the Indexer
  2. PDFs are not consistent, we can not use simple text extractor and process them. we have to render every page and send to VLM.
  3. Websites can be screen-shotted, but sum info is hided behind collapsables, buttons, etc. hence we introduced browser accessability tree capture.
  4. the tree is too large often times and exceeds context window of LLM. hence we introduces smart Pruning of the tree.

  5. Agent looses context in subsequent prompts for extracting per-page data and building knowledge base. hence we introduced "Context Prefix" which mentions brief summary of whats extracted so far. and agent references it in every call.

  6. Designing Retrieval

  7. simple KB traversal along with vector search worked, but it did not cover every use case properly. it often missed important keywords, or took too many traversal steps to reach to fine-grained key words. Hence we introduced classic BM-25 search, significantly improving the flow, both in accuracy and token consumption and response latency.

  8. Too many internal tools bloat the context window of LLM agent. solved by having classes of tools and providing only core tools directly, all the widgets tools were provided in separate agent call - significantly improving LLM responses, and reducing processing time

  9. Obsidian Like Interface

  10. While Obsidian is a good starting point, we have much more than that. Folder hierarchy along with the MD file mesh, and PDF links, Images links, etc. the UX was still to be figured out.

  11. Annotations in PDFs, Images, and md files having cropped images (VLLM bounding box), all of these were hard to glue up.

Accomplishments that we're proud of

The product works exceptionally well for our use cases. to the point that If we come across a research paper, or informative website, we directly dump it to MEMUX, and later chat with it about questions, finding specific formulae, and even reading md files. It gives a much better experience than say directly dumping the paper to chatgpt, or similar app, because the Knowledge base is persistent and we can reference the same thing in future conversations and get same accuracy and depth.

What we learned

  • Knowledge re-discovery is a silent problem that no one notices until it is solved for them. We unknowingly spend about 30 mins to 2.5 hours (in case of knowledge workers like researchers, students, and early professionals) a day, which would otherwise be invested in productive work.

  • Building with agentic tools looks easy, but is way more challenging when we try to get something useful out of it.

  • Sure if we would have built MEMUX without agentic coding tools, it would have taken ~ 6 Months to 1 Year. but with ai coding tools, it took 2 - 3 months.

What's next for MEMUX

The next phase of MEMUX is teams. Teams build up a massive knowledge base in the form of WhatsApp chats, Slack, mails, etc. MEMUX will provide a group chat where teams build their knowledge base smartly with the help of AI, thereby increasing productivity at the time of re-discovery, report generation, and auditing.

We have also planned subscription tiers & limits for individuals, team, and enterprise users. free tier gives generous inference of Gemma4 31B, with 2 GB cloud backup. pro tier will provide frontier models and extended limits for Gemma4 31B. team plans: free for upto 5 members, limited inference and cloud storage. pro for upto 20 members and, enterprise for 20+ members.

Built With

  • cloudflare-do
  • cloudflare-r2
  • gemma4-31b
  • google-aistudio
  • indexeddb
  • next.js
  • opfs
  • pg-vector
  • pglite
  • sqlite
  • tailwindcss
  • typescript
  • vite
  • zustand
Share this project:

Updates