-
-
Archive Screen
Loom — An AI Learning Companion That Points, Not Just Explains
Inspiration
I have always believed that the biggest problem in learning is not access to information — it is access to the right information at the right moment. The internet is drowning in tutorials, videos, articles, and documentation. Yet somehow, a person who wants to build something from scratch still ends up lost, overwhelmed, or stuck watching a 45-minute video hoping the answer to their specific question appears somewhere in the middle.
The idea for Loom came from a simple frustration: every AI tool I had used could explain things, but none of them could point. None of them could say — "go to 3:06 in this specific video, watch until 4:45, and that is exactly what you need right now." None of them understood that a beginner building a birdhouse and an engineer designing a PCB both need the same thing — a guide that meets them exactly where they are, with resources scoped precisely to the task in front of them.
I also thought about how a great human mentor works. They do not hand you a reading list. They sit with you, ask what you already know, break the work into pieces, watch you struggle just long enough to learn something, and step in at exactly the right moment. Loom is my attempt to build that mentor in software — for any project, in any discipline, for any person.
What I Built
Loom is a multi-agent AI learning companion that guides users through any project — woodworking, music, programming, cooking, electronics, or anything else — by breaking it into structured phases and tasks, and finding the exact resources needed to complete each one.
The key differentiator is resource precision. When Loom finds a YouTube video, it does not just return a link. It fetches the full transcript, chunks it into segments, embeds each chunk using a sentence transformer model, stores the vectors in MongoDB, and runs a semantic similarity search to find the specific moment in the video most relevant to the user's current task. The result is a timestamped deep link — youtube.com/watch?v=...&t=186s — that drops the user at the exact second they need.
The same principle applies to articles. Loom scrapes the page, chunks it by paragraph, embeds and stores the content, and surfaces the single most relevant paragraph inline — so the user never has to leave the app or lose their focus.
Architecture
The backend is built on Google ADK with five specialized agents:
- Onboarding Agent — collects the user's name, goal, domain, experience level, and preferred resource types through a warm conversational flow. It includes a verification step where it asks a domain-specific follow-up question to calibrate the user's actual experience level rather than trusting the label they give themselves.
- Planning Agent — generates a project roadmap as a flat list of phases, then produces 3–7 specific, actionable tasks for the active phase. Tasks are tailored to the user's experience level — a beginner gets smaller, more detailed steps; an advanced user gets broader ones.
- Resource Agent — uses the YouTube pipeline and DuckDuckGo MCP to find and retrieve resources for each task. It never surfaces a whole video — only the relevant segment.
- Progress Agent — tracks task completion and handles stuck users with three escalation levels: rephrase the task, find different resources, or break the task into smaller subtasks.
- Root Agent — orchestrates the full flow using ADK's
SequentialAgentandLoopAgentpatterns, owns the MongoDB MCP connection for project persistence, and coordinates handoffs between agents.
The agent pipeline follows this structure:
Onboarding → [Planning → [Resource → Progress] × tasks] × phases
Loop controllers check completion conditions and escalate to break loops when phases or projects are finished.
MongoDB plays a central role in two distinct ways:
Vector search —
transcript_chunksandarticle_chunkscollections in MongoDB Community Edition 8.2+ use$vectorSearchwith atranscript_vector_indexto find semantically relevant content. Embeddings are generated locally usingsentence-transformerswith theall-MiniLM-L6-v2model (384 dimensions, cosine similarity).Project persistence — full project documents including phases, tasks, resources, and progress are saved and loaded via the MongoDB MCP server, giving users a persistent record of their learning journey across sessions.
The API layer is built on FastAPI wrapping ADK's built-in get_fast_api_app(), adding custom REST endpoints for project management and a WebSocket endpoint that streams typed Loom events to the frontend in real time:
text | tool_call | tool_result | state_delta | agent_switch | project_ready | done | error
The frontend is built in Kotlin Multiplatform targeting wasmJs (web) and Android, using Compose Multiplatform, Ktor for networking and WebSocket streaming, and Navigation3 by JetBrains for adaptive multi-pane navigation. The workspace screen uses a three-pane adaptive layout — navigation drawer, chat and task pane, and resource detail pane — where YouTube videos are embedded directly in the app starting at the relevant timestamp, and article excerpts are shown inline, keeping the user fully focused without context-switching to another tab.
The Semantic Retrieval Pipeline
The core technical innovation in Loom is the resource retrieval pipeline. Here is how it works mathematically:
Given a task description $q$ and a transcript or article chunked into segments ${c_1, c_2, \ldots, c_n}$, each chunk is encoded into a dense vector:
$$\vec{v}_i = \text{Encoder}(c_i) \in \mathbb{R}^{384}$$
The query is encoded similarly:
$$\vec{q} = \text{Encoder}(q) \in \mathbb{R}^{384}$$
MongoDB's $vectorSearch finds the top-$k$ chunks by cosine similarity:
$$\text{sim}(\vec{q}, \vec{v}_i) = \frac{\vec{q} \cdot \vec{v}_i}{|\vec{q}| |\vec{v}_i|}$$
The top 3 results are sorted by their original timestamp, and the range $[\text{start}{\min}, \text{end}{\max}]$ is returned as the recommended viewing window. This means Loom does not guess — it mathematically finds the most semantically similar moment in any piece of content to the user's current learning need.
Challenges
Getting MongoDB vector search running locally was the first major challenge. The standard mongod process does not support $vectorSearch — it requires the Atlas runtime, even locally. Setting up the Atlas CLI local deployment, connecting to the correct port (directConnection=true was critical), creating the vector search index with the right filter fields, and getting Python to connect to it correctly took significant debugging. The error messages were cryptic, the ports conflicted, and the index had to be rebuilt several times.
Semantic precision was harder than expected. Early versions returned transcript chunks that were topically related but too scattered in time — a 3-minute range spanning multiple unrelated segments. The solution was to fetch the top 3 closest chunks, sort them by timestamp, and return the contiguous window. This gave users a focused, watchable segment rather than a scattershot collection of moments.
Token management became a real constraint during development. Long sessions with multi-agent pipelines exhaust context windows quickly. This prompted me to design a three-tier memory architecture inspired by how the human brain handles different types of memory — procedural memory baked into agent instructions (zero cost), episodic memory as a rolling conversation summary (compressed), and semantic memory retrieved from MongoDB on demand (only what is relevant right now). This is partially implemented and will be fully built in version two.
Streaming agent events through WebSocket required mapping ADK's raw event objects to clean, UI-consumable Loom event types. ADK events carry partial text, function calls, function responses, state deltas, and escalation signals all in the same structure. Writing map_adk_event() to correctly identify and classify each event type — and surfacing them to the right UI pane in real time — was fiddly but essential to the live, transparent experience Loom aims for.
Scope management was perhaps the hardest challenge of all. Loom naturally wants to be more — a PDF retrieval system, a deep web crawler, a podcast companion, a social platform for builders. Every session surfaced a new compelling direction. Shipping a focused, working version one required deliberate restraint.
What I Learned
I learned that the hardest part of building an AI product is not the AI — it is the plumbing. Getting a language model to generate a task list is trivial. Getting the right video segment to appear in the right pane at the right moment, reliably, with accurate timestamps, without burning a token budget, while keeping the user's session state consistent across a multi-agent pipeline — that is where the real engineering lives.
I learned that vector databases are not magic, but they are genuinely useful. Cosine similarity over transcript embeddings finds relevant content in ways that keyword search simply cannot. A user asking about "attaching components to a circuit board" will find a video segment about "soldering LEDs to a PCB" — because the semantic meaning is close, even though no keywords overlap.
I learned that agent orchestration requires patience and humility. Multi-agent systems behave in unexpected ways. Agents hallucinate tool calls. Loops do not break when you expect them to. State deltas arrive out of order. Building reliable agent pipelines means designing for failure, not just for success.
I learned that constraints are creative. The competition deadline forced prioritisation that made the product sharper. Every feature that did not ship became a clearer, better-defined version two idea.
What's Next
- Deep crawling and PDF retrieval — a unified pipeline that follows links, downloads PDFs, extracts content, and goes as deep as the content goes
- Brain memory model — full implementation of episodic compression and semantic retrieval to solve the token exhaustion problem
- Dynamic resource playlists — stitching timestamped clips from multiple videos into a single ordered learning sequence
- Podcast resources — the same semantic pipeline applied to audio content
- Loom for builders — a social layer where projects become shareable objects, roadmaps can be forked, and the building process itself becomes community knowledge
Log in or sign up for Devpost to join the conversation.