ContextLoop

Inspiration

I wear Bee during mock interviews, study sessions, and technical conversations. It transcribes everything. And every time I reviewed the transcript the next morning, I noticed the same pattern: I'd say something like "I never quite get how consistent hashing avoids moving all the keys" - and then move on. The conversation kept going. I meant to look it up. I never did.

The problem isn't motivation. It's friction. Between the moment of confusion and the moment of learning, there are too many steps: remember the topic, find a source, read it, check your understanding. By the time you sit down to study, you've forgotten what specifically confused you and why.

Bee already captures the moment. It just doesn't close the loop.

ContextLoop is our attempt to close it - automatically, using exactly what Bee heard, grounded in real sources, delivered on the screen where I actually relax in the evening: Fire TV.

What We Built

ContextLoop is a full end-to-end learning pipeline:

Wear Bee during a conversation. Say something that reveals a gap - imprecise, in-context, real.
Run make sync that evening. ContextLoop connects to Bee's MCP server over stdio, reads recent conversations, and finds gaps in your own words - not summaries, not keywords, but the 25-word quote from the exact utterance that revealed confusion.
The worker on EC2 grounds each gap in Wikipedia and trusted documentation. It calls Amazon Bedrock (Claude Haiku 4.5) to write a structured lesson - 4 to 6 beats - and a shorter alternate from a different angle. Amazon Polly narrates it with sentence-level speech marks, which drive caption timing. ffmpeg assembles a captioned MP4 with burned-in subtitles.
The Fire TV app (built with React Native for Vega) shows three rows: Tonight, Revisit, and Not confident yet. A lesson opens with a gap card showing what you said, what you knew, and what was unclear. The video plays with full remote control support - play/pause, 5-second seek, 0.5× to 1.25× speed. After the video, a QR code screen shows every source with a 15-second countdown, so you can open them on your phone.
The quiz asks one multiple-choice question. A correct answer writes a permanent fact back to Bee's memory: "Explained consistent hashing correctly on Oct 21." A wrong answer writes a Bee todo with a 7 pm reminder the next day.

The full loop: Bee hears your confusion => ContextLoop teaches it => Bee remembers you learned it.

How We Built It

The Mac sync job is Python 3.12 with the official mcp SDK connecting to bee mcp serve over stdio. It calls six Bee MCP tools: bee_list_conversations, bee_get_conversation_transcript, bee_search_conversations, bee_create_todo, bee_create_fact, and bee_list_facts. Every stage runs as a standalone Python module callable from the command line, which made debugging individual pipeline stages fast.

The backend is FastAPI + SQLite running in Docker Compose on a single EC2 t3.small, behind CloudFront for HTTPS without a domain. The lesson worker runs in the same Compose stack. The deploy is plain AWS CLI scripts - idempotent, ships what you've committed, no Terraform or CDK.

Amazon Bedrock runs five separate prompts: triage, gap detection, history enrichment, grounding, and lesson writing. The grounding step is fully automatic - sources.yaml lists trusted domains; the model proposes pages, the code fetches them, and a grader scores how well the passages actually cover each unclear point. A lesson that can't be grounded at 0.5 or above is refused with a specific reason rather than silently made up.

Amazon Polly generates narration with sentence-level speech marks - SSML sentence boundaries - which are used to generate per-sentence caption timing. Every lesson has accurate burned-in captions synchronized to the audio.

The Fire TV app is React Native for Vega (TypeScript), tested in the Vega Virtual Device. It handles the Fire TV remote's full key set: play/pause arrives as play not playpause, OK from the on-screen remote arrives as kpenter not enter, and the W3C media player rejects HTTP URLs silently regardless of the cleartext policy - all worked around in production code.

The mobile app (React Native / Expo) serves as a companion, showing the same lesson cards and quiz flow for users who prefer their phone.

Challenges

The speaker labeling problem. 60–90% of Bee utterances arrive with speaker: "Unknown". In one real conversation - 368 utterances - only 39 were labeled with my name. Detection can only confidently use named lines, so a large fraction of real gaps are invisible to the system. The workaround is to use Unknown lines as context but not as gap sources. This is the single biggest limit on the system's recall, and it's entirely outside our control.

Grounding without hand-picking links. Early versions required manually providing Wikipedia URLs per topic. The current system is fully automatic - the model proposes pages, the code fetches them, and the grader decides whether they actually cover the unclear points. On the first six real threads, three became lessons and three were refused with reasons. One false "ready" (a capital-markets article for a data-pipeline question) was caught by a grading rule. A lesson that keeps failing is retried a bounded number of times before being parked - never fabricated.

Privacy enforced in code, not policy. A quote on screen can never be invented: the code copies the exact line from the transcript by line ID. Numbers of 6+ digits and email addresses are stripped before anything leaves the Mac. A dedicated test pushes a fixture full of names, an account number, a street address, and a medical record through the entire pipeline and asserts none of them reach SQLite, the slide HTML, the narration, the logs, or Bee. It found two real leaks during development, both fixed.

The new AWS account SCP problem. The new-experience AWS sign-up silently restricts the account to one region via a service control policy. First calls to Polly and S3 failed with an opaque explicit deny in a service control policy error that didn't name the restriction or link to a fix. Polly's neural engine isn't available in that region. After activating "advanced features" (irreversible, removes spend limits), a second SCP still denied Polly in us-east-1. The fix required editing the organization policy directly - adding "polly:*" to the exceptions list - which used 5,097 of 5,120 allowed characters. This cost several hours and is documented in full in FRICTION.md.

The write-back idempotency problem. bee_create_fact has no idempotency key. If the process crashes between creating a fact in Bee and recording its ID locally, the next run creates a duplicate. bee_search_facts doesn't find a fact created seconds earlier, so an exact-match check isn't possible. The workaround scans bee_list_facts newest-first and stops at the first fact older than the event being recorded.

Making the Fire TV remote work reliably. The on-screen OK button sends kpenter, not enter. The dedicated play/pause key sends play, not playpause. The W3C media player silently rejects HTTP URLs even with a cleartext policy, firing MediaError.code = 4 with an empty message. JavaScript console.log output doesn't appear in loggingctl on the Virtual Device. Each of these required a workaround discovered from system logs or community posts, and each is documented.

What We Learned

I learned that closing a loop is harder than opening one. Reading from Bee is straightforward. Writing back to Bee in a way that's idempotent, deduplicated, and actually useful required thinking carefully about failure modes that don't show up until production.

I learned that grounding is the hardest part of any lesson-generation system. It's easy to generate plausible-sounding text. It's hard to verify that the text is actually supported by evidence and actually covers the specific thing the learner didn't understand. The grading step - scoring passage coverage per unclear point - is the most important piece of code in the whole project.

I learned that the wearable matters. The input that makes this system different is not a search query or a reading list. It's what I actually said - imprecise, real, in-context. That signal is only available because Bee was worn. No other interface captures the moment of confusion in the words of the person who was confused.