Scout & Capy

A personal brain that knows what it doesn't know.

Comment

Scout & Capy

A personal brain that knows what it doesn't know.

Capy is memory: a Cognee knowledge graph built from your own email, docs, notes and calendar. Ask it something and it first enumerates what it structurally cannot answer — a person with no attributes, a job title last observed 428 days ago, two entities with no path between them — and compiles each hole into one concrete search query. Scout fills exactly those holes through Bright Data, folds the result back in as cited, untrusted-tagged graph data, and re-answers. A Strands agent runs the loop.

It runs with no API keys at all.

Demo video

Watch on YouTube — 2 minutes, narrated. Also in the repo as demo/scout-and-capy-demo.mp4.

A single unbroken screen recording of the real app against the real backend; nothing on screen is a mock-up. The run in the video reports 5 gaps · 3 filled · 2 live, and the live Bright Data results are readable in the trace rail at about 1:28. See demo/README.md for how it was made and for the one claim the narration makes that the footage does not show directly.

The idea

A vector store cannot tell you what it doesn't know. Ask it anything and it returns k chunks with confident similarity scores — the absence of an answer looks exactly like the presence of one. "I don't know" is not a shape a top-k retrieval can produce.

A knowledge graph makes absence structural, and therefore enumerable: a Person node with zero descriptive attributes; a volatile attribute whose observed_at is 14 months old; two entities the question needs joined with no asserted path between them. Those are queries over topology, typing and time — which is the reason to have a graph rather than an index.

And because each hole is a specific missing fact, it compiles to a specific fetch. The web call is not decoration bolted onto a graph demo; the graph states its own holes and Scout fills those holes, nothing more:

recall  →  detect gaps  →  fetch exactly those  →  ingest  →  re-answer
Cognee     Cognee           Bright Data             Cognee    Cognee
                        ← ← ← Strands picks the order ← ← ←

Every gap becomes a FetchPlan(query, expects, attach_to). Every fetched fact lands tagged node_set="web", trust="untrusted", with a source URL and a fetch time. Web text becomes graph data — node attributes and citations — never agent instructions.

What each sponsor technology does here

Technology What it does here API surface used
Cognee 1.6.0 Capy. Builds the graph from the private corpus, answers over it, reports which graph elements the answer used, and rewires on feedback. add(node_set=) · cognify(ontology_file_path=, temporal_cognify=) · search(query_type=SearchType.GRAPH_COMPLETION, session_id=, include_references=, feedback_influence=, top_k=15) · get_session() → used_graph_element_ids · add_feedback() + memify() · visualize_graph_json() · get_schema_inventory() / validate()
Bright Data Scout. Fetches exactly the gaps the graph named, over a transport that needs no zone provisioning, with a provenance stamp on every result. Remote MCP: search_engine, scrape_as_markdown · alternate POST https://api.brightdata.com/request (SERP brd_json=1, Unlocker data_format: "markdown")
Strands 1.56.0 The agent that decides which move comes next, and streams every tool call into the UI. Agent(model=, tools=, system_prompt=) · five @tool functions · agent.stream_async(q, limits=Limits(max_iterations=14)) · strands.tools.mcp.MCPClient(url=) · OpenAIModel / AnthropicModel

Cognee — the memory layer (app/services/memory.py)

  • Ingest, tagged by origin. cognee.add(texts, dataset_name="scout_and_capy", node_set=[tag]), one tag per source (email, docs, notes, calendar) and web for everything Scout brings back. The tag is what lets the UI colour web-derived knowledge differently and what keeps it queryably separable from the user's own corpus.
  • Cognify with a shape. cognee.cognify(datasets=[…], ontology_file_path="ontology/scout_capy.owl", temporal_cognify=True); web ingest adds incremental_loading=True. The ontology's job is to keep the graph a shape rather than a hairball — works_at is one edge type, not nine synonyms.
  • The ontology is what makes staleness structural. ontology/scout_capy.owl declares 7 top-level classes (Person, Organisation, Project, Meeting, Document, Claim, Topic) plus 9 specialisations, 16 object properties and 10 data properties. Six data properties are volatile — jobTitle, employer, headcount, fundingStage, pricing, version — and observedAt is the field the staleness check reads. Naming volatility in the ontology is what turns "is this stale?" into a query instead of a guess.
  • Retriever-reported provenance, not inferred. search(..., session_id=…, include_references=True) makes the retriever record used_graph_element_ids = {node_ids, edge_ids} on the session's QA entry. The app reads it straight back — from cognee.api.v1.session import get_session; await get_session(session_id=…, last_n=1) — and emits those exact ids as a highlight event, so the pond lights up the elements Cognee actually used. Nothing is re-derived or guessed.
  • Feedback that steers, not feedback that is filed. add_feedback(session_id=, qa_id=, feedback_text=, feedback_score=) → cognee.memify(dataset=), and once a rating is accepted, subsequent searches pass feedback_influence=0.35. Without that last step the correction is recorded and ignored, which is the quiet failure mode of most "we have a feedback loop" demos.
  • Renderer payload. from cognee.api.v1.visualize import visualize_graph_json → {"nodes": […], "links": […]} (max_nodes=900), with get_graph_engine().get_graph_data() as the fallback. Both shapes normalise into one GraphView, so gap detection and the frontend have a single code path whether the graph came from a live cognify or from a frozen snapshot.
  • Introspection worth showing. get_schema_inventory(dataset=, samples_per_type=4) and validate(dataset=) for "what did extraction actually build, and does it match the ontology?"
  • Store: ladybug (GRAPH_DATABASE_PROVIDER) — local, zero setup, no Docker and no Neo4j to demo. COGNEE_API_URL + COGNEE_API_KEY switch memory to a managed Cognee Cloud tenant instead (app/services/cognee_cloud.py), which is preferred when configured because the tenant runs extraction and embeddings server-side and needs no LLM key of ours. Its component health and quota show at /api/health — that distinguishes "tenant down" from "our key is wrong", which are very different problems to have thirty seconds before a demo.
  • The closer, wired. .mcp.json at the repo root points cognee-mcp at the same backend/.cognee data root the app uses, so the same graph answers from inside Claude Code. The memory outlives the demo UI.

Search modes are exposed as mode=: graph → GRAPH_COMPLETION, rag → RAG_COMPLETION (the split-screen baseline), plus cot → GRAPH_COMPLETION_COT, temporal → TEMPORAL, triplet → TRIPLET_COMPLETION.

Bright Data — the retrieval layer (app/services/brightdata.py)

  • Transport 1, remote MCP (default, verified live). Streamable HTTP against https://mcp.brightdata.com/mcp?token=…: initialize (protocol 2024-11-05) → notifications/initialized → tools/call, with data: SSE frames parsed out, the mcp-session-id cached, and one automatic re-handshake on 400/401/404/409. tools/list returns, live: search_engine, scrape_as_markdown, search_engine_batch, scrape_batch, ask_brightdata_assistant. We call search_engine {query, engine: "google", geo_location?} and scrape_as_markdown {url}. It needs only an API key — no zone to provision — which is why it is the default.
  • Transport 2, REST (opt-in). POST https://api.brightdata.com/request with {"zone", "url", "format": "raw"}; SERP by appending &brd_json=1 to the Google URL, Unlocker by adding "data_format": "markdown". Requires BRIGHTDATA_SERP_ZONE / BRIGHTDATA_UNLOCKER_ZONE.
  • Transport 3, recorded fixtures. Author-curated files in snapshots/web/, then the auto-recorded snapshots/web/_cache/…; every live result is written to the cache on the way through, so a beat that worked once replays when the venue wifi dies. A curated fixture declares what it answers rather than relying on a query slug — "match": [["tomas","brandt"], ["brandt","security"]], a list of term groups, applying when every term in any one group appears in the query — because a one-word change to a compiled gap query would otherwise miss the fixture mid-demo. It may also carry extracted: [{relation, target, target_type, attrs, quote}], which lets the keyless tier write the same typed edge cognify would have derived from the page (memory.ingest_facts, which resolves the target to the existing node so the path actually forms) instead of merely attaching a document.
  • The trust boundary. Bright Data's MCP wraps page content in =====UNTRUSTED_<nonce>_BEGIN===== … =====UNTRUSTED_<nonce>_END=====. The regex is keyed on the nonce it was handed (a \1 back-reference), so fetched text cannot forge its own closing marker. The envelope is stripped at the edge and everything downstream carries trust="untrusted", node_set="web", a source_url and a fetched_at. The agent's system prompt states that web content is data and that a page asking to change behaviour is to be ignored and reported.
  • Provenance stamp. Provenance.label() renders live · search_engine · 2s ago or cached · recorded, and the UI shows it verbatim. A cached result is never labelled live.
  • Budget. SCOUT_FETCH_BUDGET=6 per process; the planner takes at most 3 gaps per question and stops below priority < 0.25. BRIGHTDATA_TRANSPORT=auto|mcp|rest|fixture|off forces a transport, and Scout.probe() reports reachability, the live tool list, remaining budget and fixture count.

Strands — the orchestrator (app/services/agent.py)

  • Agent(model=…, tools=[…], system_prompt=SYSTEM_PROMPT, name="scout-and-capy", callback_handler=None) — no callback handler because we consume the stream ourselves.
  • Five @tool functions, all thin wrappers over one shared Toolbox: recall_memory, detect_knowledge_gaps, fetch_missing, trace_path, compare_flat_retrieval. detect_knowledge_gaps hands back each gap's ready-made fetch_query, and the prompt tells the model to pass it verbatim and to fetch nothing a gap did not ask for.
  • Event mapping. agent.stream_async(question, limits=Limits(max_iterations=14)): data deltas become token events, a new current_tool_use.toolUseId becomes a "model chose a tool" step (deduped by id), force_stop becomes an error event. The Toolbox emits its own RUNNING/DONE steps, so a model-chosen tool call animates exactly like a planned one.
  • Bright Data's MCP, directly. MCPClient(url=f"{mcp_url}?token=…", prefix="bd", startup_timeout=20, continue_on_error=True) is appended to the tool list, so the agent can reach scrape_as_markdown on a specific URL when a SERP is not enough. Our own fetch_missing stays the demo path because it carries provenance and writes to the graph.
  • Explicit model providers. Strands defaults to Amazon Bedrock, which this project does not use, so the provider is always constructed by hand: AnthropicModel(client_args={"api_key": …}, model_id="claude-sonnet-4-5", max_tokens=2048) or OpenAIModel(client_args={"api_key": …}, model_id=…, params={"temperature": 0.2}).
  • Degradation. With no model key — or if the agent errors mid-flight — PlannedDriver runs the canonical sequence and emits the identical event stream. It is not a mock: same Cognee searches, same gap detector, same live Bright Data fetches. The only thing it lacks is an LLM choosing the order.

The six gap kinds

Defined in app/services/gaps.py. Ranked by priority = 0.65 · relevance + 0.35 · severity, deduped to one gap per subject (highest priority wins), top 6 returned. Relevance dominates on purpose: a severe gap nobody asked about is noise.

Kind Severity What it detects Why a vector store can't
bridge 1.00 Two entities the question joins, with no asserted path within 4 hops. "No path" is a traversal result. Top-k always returns k rows, so disconnection is not a thing it can report.
unknown 0.92 A capitalised span in the question that matches no node at all. The index returns its nearest neighbours anyway, with confident scores. This is the "a judge names a company" path.
dangling 0.88 A named entity with relationships but zero descriptive attributes — not weakly connected, unattributed. There is no chunk to retrieve: the missing information was never written down.
conflict 0.74 One subject, one volatile attribute, two different values across nodes. Both chunks rank, neither wins, and nothing flags the disagreement.
stale 0.66 A volatile attribute past its own shelf life — version 120 days, pricing 180, headcount 240, job_title 400, employer 500, location 700 (SCOUT_STALE_DAYS is the fallback for unlisted attributes). Similarity has no time axis. A 14-month-old sentence ranks exactly like today's.
undescribed 0.45 A typed node missing an attribute its class should carry (Person → job_title, employer; Topic → version; Organisation → headcount). Requires a schema. A chunk index has no notion of what a Person ought to have.

Staleness is judged per attribute, not against one global cutoff, because the rates differ by an order of magnitude: a pinned spec version is suspect after four months, a job title after a bit over a year, an office location after two. That sharpens the compiled query as well as the detection — at 428 days Priya's gap asks for job_title alone, because her employer has not reached its 500-day shelf life.

Two further details that keep the detector honest:

  • Co-mention is not a relationship. Path-finding uses path(a, b, semantic_only=True), which refuses to walk mentions-class edges — 38 of the demo graph's 95 edges. If two people named in the same email counted as connected, every entity would sit two hops from every other and "is there a path?" would always answer yes: the same false confidence a vector store gives you.
  • Relevance propagates. graph_view.relevance_map(tokens, hops=1, decay=0.55) spreads lexical relevance one hop along asserted edges. The question "who should own the Acme security review?" contains no token matching "Tomas Brandt", yet Tomas is the one node standing between that question and its answer; he earns relevance 0.73 from his neighbour. Without propagation the ranking surfaces whichever node happens to share a word and buries the one that actually blocks the answer. It is deliberately lexical, not embeddings, because gap ranking has to be explainable on stage and a cosine score is not an explanation.

Architecture

┌── frontend/ ── Next.js 16 · React 19 · pnpm ──────────────────────────────┐
│  components/pond/        Pond + pondController — 3d-force-graph / three,  │
│                          coloured by node_set (own corpus warm, web blue) │
│  components/trace/       TraceRail · StepCard · GapCard · ProvenanceRow   │
│  components/answer/      AnswerPanel · CitationChips · TripleList ·       │
│                          FeedbackBar                                      │
│  components/compare/     CompareView — graph vs flat retrieval            │
│  components/{time,status,mascots}/  TimeSlider · StatusRail (tier) ·      │
│                          Scout & Capy sprites                             │
│  hooks/useAskRun.ts      consumes the run; lib/api.ts opens the SSE       │
└─────────────────────────────┬─────────────────────────────────────────────┘
                              │  GET /api/ask/stream (SSE), POST /api/ask as
                              │  fallback. One event contract, two drivers:
                              │  step · token · graph_delta · highlight ·
                              │  answer · done · error
┌─────────────────────────────┴── backend/ ── FastAPI · uv · Python 3.12 ───┐
│  app/main.py                  app, CORS, "no traceback is the demo"       │
│  app/routes/api.py            /api/{health,graph,gaps,ask,ask/stream,     │
│                                     feedback,compare,timeline,inventory,  │
│                                     demo/script,demo/reset}               │
│  app/core/config.py           tiers, env, fetch budget, paths             │
│  app/core/events.py           Phase / Status / Step — the UI contract     │
│                                                                           │
│  app/services/agent.py        ScoutCapy · Toolbox (the 5 tools) ·         │
│                               StrandsDriver | PlannedDriver               │
│  app/services/memory.py       CogneeMemory | SnapshotMemory (one Answer)  │
│  app/services/cognee_cloud.py managed-tenant client, probe() + quota()    │
│  app/services/gaps.py         GapDetector → Gap → FetchPlan               │
│  app/services/graph_view.py   GraphView · path() · relevance_map()        │
│  app/services/brightdata.py   Scout: mcp | rest | fixture + Provenance    │
│                                                                           │
│  ontology/scout_capy.owl      the classes, relations and volatile fields  │
│  snapshots/pond.json          frozen graph — 40 nodes, 95 edges           │
│  snapshots/facts.py           authored entity core + the planted gaps     │
│  snapshots/web/               recorded fetches (curated, then _cache/)    │
│  seed/email/*.md              the private corpus (docs, notes, calendar   │
│                               directories are wired and currently empty)  │
│  tests/                       the suite: network-guarded, env-isolated    │
└──────────┬──────────────────────────────────┬─────────────────────────────┘
           │                                  │
   Cognee (ladybug, local)            Bright Data
   or Cognee Cloud, + an LLM          remote MCP  ·  POST /request
   key for cognify/search             search_engine · scrape_as_markdown

Everything above graph_view.py is written against one GraphView, so there is exactly one code path whether the graph came from a live cognify or from the frozen snapshot — and snapshots/facts.py + scripts/build_snapshot.py produce the same {"nodes", "links"} shape visualize_graph_json() returns.

Run it

Backend

cd backend
uv venv --python 3.12 .venv
uv pip install -e .
cp .env.example .env
.venv/bin/python scripts/build_snapshot.py
.venv/bin/uvicorn app.main:app --reload

Frontend

cd frontend
pnpm install
pnpm dev

Backend on :8000, frontend on :3000. The UI reads NEXT_PUBLIC_API_BASE (see frontend/.env.local.example) and defaults to http://localhost:8000. GET /api/health prints the tier, the graph stats, Scout's reachability with its live tool list, and the installed cognee / strands-agents versions; interactive docs are at /docs.

Tests

cd backend && .venv/bin/python -m pytest -q      # 122 passed

The suite is env-isolated and network-guarded — it replaces httpx's outbound entry points with something that raises, so an accidental live fetch fails loudly instead of quietly passing on the author's wifi. tests/test_planted_gaps.py is the one that matters most: it fails if a later edit gives Tomas an attribute or refreshes Priya's timestamp, which would make the gap detector go quiet and leave the demo with nothing to find.

build_snapshot.py compiles snapshots/facts.py + seed/ into snapshots/pond.json and prints the demo's invariants — Tomas Brandt's degree and attribute count, Priya's attribute age, and the hop counts between the entities the demo turns on. It fails the build if a planted gap has been broken (if Tomas gains an attribute, loses his only edge, or already has an employment edge to Helios Labs, there is no demo).

Keys are all optional:

  • No keys at all → snapshot tier. The frozen graph loads, gap detection is fully live (it is pure graph structure), fetches replay from recorded fixtures, and answers are composed by traversal. Nothing is stubbed out and nothing is blank.
  • BRIGHTDATA_API_KEY alone → scout tier. Live web fetch over the remote MCP against snapshot memory. No zone setup needed.
  • LLM_API_KEY (or Cognee Cloud creds) → live Cognee add/cognify/search and the real Strands agent choosing the moves. Both together is the full tier.

Windows (PowerShell)

Everything below the wrappers is portable — the Python uses pathlib and reads every file as explicit UTF-8, nothing shells out, and lancedb, ladybug and onnxruntime all publish Windows wheels. Only the Makefile is POSIX-only (it needs /bin/bash, trap and wait), so run its steps directly. The package manager is pnpm, not npm, in both frontend/ and demo/.

# backend — note Scripts\ rather than bin/
cd backend
uv venv --python 3.12 .venv
uv pip install --python .venv\Scripts\python.exe -e . pytest pytest-asyncio
Copy-Item ..\.env.example .env
.venv\Scripts\python.exe scripts\build_snapshot.py
.venv\Scripts\python.exe -m pytest -q
.venv\Scripts\uvicorn.exe app.main:app --reload --port 8000

# frontend, in a second terminal
cd frontend
pnpm install
pnpm dev --port 3000

Under Git Bash or WSL the Makefile works unchanged — it detects .venv/Scripts and the .exe suffix — so make setup && make dev is fine there.

Two Windows-specific notes:

  • If PYTHONUTF8 is unset and you add files to seed/ yourself, write them as UTF-8. The corpus is full of em dashes; the app always decodes explicitly, but your editor's default may not encode that way.
  • .mcp.json interpolates ${PWD}, which resolves in PowerShell, Git Bash and WSL but not in cmd.exe. Replace it with an absolute path if you use cmd.

Capability tiers

From app/core/config.py; reported at /health and printed in the UI status rail.

Tier Keys present Memory Retrieval
snapshot none frozen snapshots/pond.json + traversal answerer recorded fixtures only
scout Bright Data frozen snapshot live Bright Data fetch
full Bright Data + LLM (or Cognee Cloud) live Cognee cognify / search / feedback live Bright Data fetch

SCOUT_FORCE_SNAPSHOT=1 pins snapshot memory for a rehearsal; SCOUT_ALLOW_LIVE_FETCH=0 keeps memory live but takes Scout offline. One quirk worth knowing before you test it: the tier name tracks Scout, so an LLM key without a Bright Data key still reports snapshot. The memory engine is reported separately and independently — memory.engine is snapshot, cognee-local or cognee-cloud, derived from the same branch memory() actually takes — and that pair is what the UI status rail shows.

Demo script (~2:30)

The beats are served by GET /api/demo/script (stale · multihop · live · cold · compare), so the UI's demo buttons and this list cannot drift apart.

  1. 0:00–0:15 — the pond. Pre-built graph on screen, coloured by node_set. Tier badge visible, top right. Say what the corpus is: one founder's email, docs, notes and calendar.
  2. 0:15–0:55 — "Prep me for the Acme Robotics call." Recall answers from memory, then the gap panel fills: dangling on Tomas Brandt (0.592) and stale on Priya Raghavan (0.546) — her title and employer last observed 428 days ago, past the 180-day shelf life. Scout fetches, Capy ingests, the answer comes back with a dated, cited correction.
  3. 0:55–1:30 — "Who from Otterloop should own the Acme security review?" Top gap is dangling on Tomas Brandt at 0.779 — one relationship, zero attributes — and the question never says his name; he is surfaced by relevance propagation. One fetch supplies previously_worked_at → Helios Labs; the new edge animates in; trace_path prints "2 hop(s) (was 4)". The recommendation goes from "Dana, she owns compliance" to "Dana — she and Tomas overlapped at Helios Labs 2020–2024."
  4. 1:30–1:55 — the baseline. Same question, same corpus, flat chunk retrieval (compare_flat_retrieval → RAG_COMPLETION). It returns four plausible paragraphs and cannot join evidence across four files — not because it is implemented badly, but because ranking chunks is the wrong operation for that question.
  5. 1:55–2:15 — feedback. Thumbs-down with a correction → add_feedback + memify → feedback_influence=0.35 on the next search. Re-ask; retrieval has moved. (At snapshot tier the same beat reweights the edges that answer leaned on and says on screen that the mechanism is the cruder stand-in.)
  6. 2:15–2:30 — the closer, genuinely live. Either the live beat — "is our note on the Model Context Protocol version still current?", a real public fact our corpus pinned at 2025-06-18 — or hand over the keyboard: any company a judge names has no node, fires an unknown gap (priority 0.94) and is fetched on the spot, with the provenance stamp to prove it.

Repo layout

.
├── README.md
├── slides.html                     5-slide pitch deck, self-contained
├── plan.md                         the build plan, kept for the record
├── .env.example                    every key, all optional — copy to backend/.env
├── .mcp.json                       cognee-mcp + Bright Data MCP, for the closer
├── backend/
│   ├── app/
│   │   ├── main.py                 FastAPI app
│   │   ├── core/{config,events}.py tiers · the Step event contract
│   │   ├── routes/api.py           /api/* including the SSE run stream
│   │   └── services/{agent,memory,cognee_cloud,gaps,graph_view,brightdata}.py
│   ├── ontology/scout_capy.owl     7 classes + 9 subclasses, 16 relations
│   ├── scripts/
│   │   ├── build_snapshot.py       compiles + asserts the demo's invariants
│   │   └── seed.py                 real cognee add + cognify over seed/
│   ├── seed/email/*.md             the synthetic corpus
│   ├── snapshots/
│   │   ├── facts.py                authored entity core, planted gaps
│   │   ├── pond.json               frozen graph (40 nodes / 95 edges)
│   │   └── web/                    curated fixtures, then _cache/
│   ├── tests/                      122 tests, network-guarded
│   └── pyproject.toml              cognee · fastapi · strands-agents
└── frontend/
    ├── app/                        layout, globals, page
    ├── components/{pond,trace,answer,compare,time,status,mascots,ui}/
    ├── hooks/, lib/                useAskRun · api (SSE) · palette · types
    └── package.json                next · 3d-force-graph · three · motion

Built With

Share this project:

Updates

Submission history