concept-graph

Import papers, get a living concept graph that merges duplicate ideas across sources, answers questions with cited nodes, and lets your mentor review and steer it.

Comment

What I learned

  • To get fewer duplicates, loosen the cheap filter. My first instinct was to tighten the embedding threshold to $0.40$ for "higher quality" matches. Duplicates went up. The blocker was starving the judge: "Attention" and "Attention mechanism" sit around $0.59$ similarity, just outside a $0.41$-distance cutoff, so the judge never even saw the pair and dutifully created a second node. Recall belongs to the cheap stage; precision belongs to the expensive one. Don't make one stage do both jobs.
  • Embeddings are a blocker, not a decision. Cosine similarity is a wonderful way to narrow candidates and a terrible way to decide identity. The semantic gap between "uses X" and "is about X" is invisible to a dot product and obvious to an LLM.
  • Distributed state lies by omission. The scariest failures weren't crashes — they were silence (see below). A surprising amount of the work was making invisible states visible.
  • A pre-release framework is a moving target. Next.js 16 renamed middleware to "proxy" and made searchParams/params async; half my debugging early on was just discovering the framework had changed under me.

Challenges I ran into

  • The silent "Queued" trap. The worker is a separate process. When it dies, every upload piles up as pending forever with nothing to drain it — and the UI just spins, indistinguishable from "still working." Fix: the worker heartbeats a key into Redis every 15s; the API exposes /api/health/worker; the dashboard polls it and shows a "worker offline" banner. A stuck job now states its real cause instead of lying.
  • The NoSuchKey race. Upload is three steps: create a pending row, browser PUTs bytes to S3, then enqueue ingestion. Early on, a big or slow upload would fail with "file not found" — the worker had raced ahead of the bytes. Fix: ingestion is only enqueued after the S3 PUT confirms.
  • A tokenizer that froze the whole server. tiktoken lazily downloads its vocabulary on first use — a synchronous call that, behind a slow/blocked network, froze the entire async event loop and hung the worker indefinitely. Fix: vendor the vocab into a local cache so the download never happens at runtime.
  • Garbage concepts from PDF guts. PyMuPDF's layout engine helpfully OCRs chart axes and figure legends into the text stream, and no parser flag suppresses it. Left alone, "Figure 3" and axis labels became concepts and polluted the embedding space. Fix: a cleaning pass strips extraction artifacts before anything is extracted or embedded.
  • Ghost nodes from split transactions. Creating a concept node and writing its provenance are separate transactions, so an interrupted ingest could strand a node that no document mentions. Fix: every ingest and every delete runs an orphan-reconciliation sweep (manual nodes exempt), so a failed import never leaves a node with no source behind it.
  • You can't put a bearer token on an EventSource. Streaming RAG answers over SSE meant the browser couldn't attach an auth header. Fix: a Next.js route handler reads the Cognito token from an http-only cookie server-side and relays the backend's SSE stream verbatim — keeping the token off client JS entirely.

What's next

Richer mentor primitives (assigning directions, reviewing diffs of the graph over time), multi-user real-time presence on the canvas, and pushing the MCP memory layer further so an external agent and the in-product agent share one brain.

Built With

Share this project:

Updates

posted an update —

I updated only the cover/thumbnail image on my project's Devpost portfolio page. I have not made any changes to the actual submission materials (text description, demo video, architecture diagram, links, etc.).

Log in or sign up for Devpost to join the conversation.