ctrl+teach

Inspiration

I kept losing the same fight.

I'd be halfway through a course, hit a doubt, and switch tabs to ask a chatbot. It would hand me back four dense paragraphs. Somewhere in paragraph two, a new doubt would appear — so I'd ask again, and the answer I hadn't finished reading would scroll up and out of view. I'd end up with more text than I started with and less understanding. The tool that was supposed to unblock me had quietly become the thing blocking me.

Then came the worse part. I'd finish the course, sit down to actually do the thing, and freeze — not because I didn't understand the concept, but because I had never seen the screen before. I knew what a CI pipeline was. I had no idea where the button was. Every time, I wished someone were sitting next to me for those ten minutes.

That's the gap Ctrl+Teach was built to close. Not "more content." Content that follows you into the work.

What it does

Ctrl+Teach is three things that are usually three different products:

1. Course generation from a prompt — that asks you back. You describe what you want to master in plain words. Then, instead of guessing, it asks a few follow-up questions: what you already know, why you need this, what you're trying to build. Most tools take your prompt and generate. Ctrl+Teach takes your prompt and interviews you first. Minutes later you have a real course — modules, lessons, generated visuals that actually explain the idea, cited sources under every claim, and a knowledge check at the end of each module — built for your starting point, not the average person's.

2. Live Classroom. Some people don't learn by reading. So the same course you just generated — the same modules, the same lessons — can be taught to you instead. A realtime voice model walks you through it conversationally, so a doubt costs you one sentence: not a tab switch, not four paragraphs, not losing your place. It's the same material either way. You choose whether you read it or sit through it with someone. And it doesn't just explain; it teaches for retention, using the Made to Stick principles — concrete examples over abstractions, a surprise that breaks your existing mental model before the correct one goes in, and stories instead of definitions. You don't leave the session having heard it. You leave having kept it.

3. Tars. A Chrome extension that comes with you into the real tool. When you're stuck on the actual screen, Tars sees what you see and guides you through it — live, in context, on your own work.

The through-line: you shouldn't have to leave what you're learning to ask about it, and you shouldn't have to leave what you learned to use it.

How we built it

Backend. FastAPI with SQLAlchemy over SQLite. A set of agents handle course generation, lesson authoring, quiz construction, and the realtime teaching session, with MCP servers wired in for Calendar and Gmail so the system can act in the learner's real world, not just talk about it.

Course generation. The prompt alone is never enough — "teach me Git" means something completely different to someone who has never made a commit than to someone drowning in merge conflicts. So a clarifying pass asks targeted follow-ups first, and those answers become the constraints the rest of the pipeline is built on. A generation model drafts the structure — modules, lessons, learning objectives — then per-lesson passes write content with source citations attached, and an image model produces the visuals. The result isn't a wall of text; it's something that looks like a course someone made for you on purpose.

Live Classroom. A realtime speech model over a WebSocket, holding the teaching state and the lesson context, so the conversation is grounded in the course you're actually taking rather than starting from zero every message.

Teaching for retention, not coverage. Most AI tutors optimize for answering. We optimized for what survives the session. The Live Classroom system prompt is built on the Made to Stick principles — lead with the concrete, break the learner's wrong mental model before installing the right one, carry ideas in stories rather than definitions, and close the curiosity gap the learner opened rather than the one the syllabus planned. A chatbot's job ends when the answer is correct. A teacher's job ends when the learner still has it tomorrow.

Tars. A Chrome extension streaming your screen to the backend, with the model annotating directly over the page.

The hardest problem: knowing where

This is the part that nearly broke the project.

Asking a realtime model to draw on your screen sounds simple. It isn't. The model is excellent at deciding what should be highlighted — "the thing you're looking for is the branch selector." It is unreliable at deciding where that is, in exact pixel coordinates, on your specific screen, at your specific zoom level. Early builds drew confident arrows pointing at nothing. That's worse than no help at all: a wrong arrow doesn't just fail, it actively misleads.

So we split the decision in two.

• The realtime model decides what to point at, in words.
• A dedicated computer-use pass resolves where that thing lives in the actual screenshot's pixel space.

Coordinates from the realtime model are never accepted and never forwarded. The system that's good at meaning does meaning. The system that's good at pixels does pixels. Separating those two jobs is what turned Tars from a demo into something you'd actually trust on your screen.

Challenges we ran into

• Latency vs. correctness. A tutor that's right in four seconds feels broken. Every added grounding step is another moment the learner sits waiting, so most of the work was making the accurate path fast rather than making the fast path accurate.
• Grounded generation. A course confidently teaching something false is more damaging than no course. Citations aren't decoration here — they're the guardrail.
• Screen understanding across real UIs. Dark mode, zoom levels, scroll position, custom themes. The screen a user is looking at is never the screen you tested on.
• Making it feel like a person, not a panel. Interrupting your tutor mid-sentence has to be normal. If asking a question feels expensive, learners stop asking — which is exactly the failure we started out trying to fix.

What we learned

Splitting a task by what each model is actually good at beats making one model do everything. The what / where split wasn't an optimization — it was the difference between a feature and a liability.

Understanding and capability are different skills. Courses teach the first and quietly assume the second. That assumption is where most learning dies — not in the lesson, but in the ten minutes after it, in front of a screen nobody prepared you for.

Learning is a conversation, not a download. The moment asking a question costs you your place, you stop asking questions.

What's next

Neurodivergent learners in K-12. Ctrl+Teach already adapts how a course is taught; the next step is adapting that delivery to how each child learns — shorter pacing for ADHD, visual or voice-first explanations for dyslexia, and clearer, more predictable instruction for autistic learners. Same standards, same rigor, different ways in. We’d build it with special educators and families, with one goal: stop asking whether a student can keep up with how we teach, and start asking whether we can teach the way they learn.

Built With

• browser-extension
• computeruse
• excalidraw
• fastapi
• mcp
• openai
• react
• websockets