Ctrl+Teach
Learning that follows you into the work.
Inspiration
I kept losing the same fight.
I would be halfway through a course, hit a doubt, and switch tabs to ask a chatbot. It would hand me four dense paragraphs. Somewhere in paragraph two, a new question would appear, so I would ask again—and the answer I had not finished reading would scroll out of view. I ended up with more text than I started with and less understanding. The tool that was supposed to unblock me had quietly become the thing blocking me.
Then came the harder part. I would finish the course, sit down to actually do the thing, and freeze—not because I did not understand the concept, but because I had never seen the screen before. I knew what a CI pipeline was. I had no idea where the button was. Every time, I wished someone could sit beside me for those ten minutes.
That is the gap Ctrl+Teach was built to close: not more content, but learning that follows you into the work.
What it does
Ctrl+Teach brings together three experiences that usually live in three different products.
Personalized course generation
You describe what you want to master in plain language. Instead of immediately guessing what you need, Ctrl+Teach interviews you first: what you already know, why you need the skill, and what you are trying to build.
Those answers shape a complete course for your actual starting point—not the average learner's. The result includes structured modules, focused lessons, generated visuals, cited sources, and knowledge checks at the end of each module.
Live Classroom
Some people do not learn best by reading. Any generated course can also become a live, conversational class. A realtime voice model teaches the same modules and lessons, grounded in the material the learner is currently studying.
A doubt now costs one sentence—not a tab switch, four paragraphs, and your place in the lesson.
The classroom is designed for retention, not just explanation. It applies Made to Stick principles: concrete examples instead of abstractions, moments of surprise that challenge an incorrect mental model, and stories instead of isolated definitions. The goal is not merely for learners to hear an answer, but to remember and use it.
Tars
Tars is a Chrome extension that follows the learner into the real tool. When someone gets stuck on the actual screen, Tars sees what they see and guides them live, in context, on their own work.
The idea connecting all three experiences is simple: you should not have to leave what you are learning to ask about it, and you should not have to leave what you learned to use it.
How we built it
Backend
The backend is built with FastAPI, SQLAlchemy, and SQLite. Specialized AI agents handle course planning, lesson authoring, quiz construction, and realtime teaching. MCP integrations for Google Calendar and Gmail let the system connect learning to the learner's real world instead of only talking about it.
Course generation
A prompt alone is rarely enough. “Teach me Git” means something entirely different to a person who has never made a commit and to someone struggling with merge conflicts every week.
Ctrl+Teach begins with a clarification pass that asks targeted follow-up questions. Those answers become constraints for the rest of the generation pipeline. A model creates the course structure—modules, lessons, and learning objectives—before lesson-level passes produce grounded content with source citations. An image model then creates supporting visuals.
The result is not a generic wall of text. It feels like a course someone made for you on purpose.
Live Classroom
The Live Classroom uses an OpenAI realtime speech model over a WebSocket connection. It maintains teaching state and lesson context throughout the session, keeping every conversation grounded in the course the learner is actually taking instead of starting from zero after each question.
Its teaching instructions optimize for what survives the session: lead with the concrete, challenge the wrong mental model before installing the right one, carry ideas through stories, and close the curiosity gap opened by the learner—not merely the one planned by the syllabus.
A chatbot's job ends when the answer is correct. A teacher's job ends when the learner still has it tomorrow.
Tars
The Tars Chrome extension captures the learner's current screen and streams it to the backend. The system interprets the interface, locates the relevant UI element, and draws guidance directly over the page so the learner can act without leaving their task.
The hardest problem: knowing where
This was the part that nearly broke the project.
Asking a realtime model to draw on a screen sounds simple. It is not. The model is excellent at deciding what should be highlighted: “the thing you need is the branch selector.” It is much less reliable at deciding where that element is in exact pixel coordinates on one particular screen, at one particular zoom level.
Early versions drew confident arrows pointing at nothing. That is worse than no help at all: an incorrect arrow does not merely fail; it actively misleads.
We solved this by separating the decision into two jobs:
- The realtime model decides what to point at, using natural language.
- A dedicated computer-use pass resolves where that target appears in the screenshot's pixel space.
Coordinates produced by the realtime model are never accepted or forwarded. The system that is good at meaning handles meaning; the system that is good at pixels handles pixels. That separation turned Tars from a demo into something a learner could trust on their screen.
Challenges we ran into
Latency versus correctness: A tutor that is correct after four seconds can still feel broken. Every grounding step adds waiting time, so we focused on making the accurate path fast.
Grounded generation: A course that confidently teaches something false is more damaging than no course. Citations are not decoration; they are a guardrail.
Understanding real interfaces: Dark mode, browser zoom, scroll position, responsive layouts, and custom themes mean the screen a learner sees is rarely identical to the screen we tested.
Making the tutor feel conversational: Interrupting a teacher mid-sentence should feel normal. If asking a question feels expensive, learners stop asking—the exact failure we set out to fix.
Accomplishments that we're proud of
We built one connected learning loop instead of three disconnected demos: Ctrl+Teach can turn an ambiguous goal into a personalized course, teach that same material in a live voice classroom, and continue helping when the learner enters the real interface to apply it.
We are especially proud of the what/where architecture behind Tars. Refusing to trust convenient but unreliable coordinates required more engineering, but it made the guidance meaningfully safer and more useful.
We are also proud that personalization begins before generation. The system does not simply produce more content; it first learns enough about the learner to create the right content.
What we learned
Splitting a task according to what each model is genuinely good at beats forcing one model to do everything. The what/where split was not a minor optimization—it was the difference between a useful feature and a liability.
We also learned that understanding and capability are different skills. Courses teach the first and quietly assume the second. That assumption is where much of learning breaks down—not in the lesson, but in the ten minutes afterward, in front of a screen nobody prepared you for.
Most importantly, learning is a conversation, not a download. The moment asking a question costs learners their place, they stop asking questions.
What's next
Our next focus is supporting neurodivergent K–12 learners. Ctrl+Teach already adapts how a course is taught; the next step is adapting delivery to how each child learns: shorter pacing for learners with ADHD, visual or voice-first explanations for learners with dyslexia, and clearer, more predictable instruction for autistic learners.
The standards and rigor should remain the same while the paths into the material become more flexible. We want to build this alongside special educators and families, guided by one question: instead of asking whether a student can keep up with how we teach, can we teach in the way that student learns?
Built with
- Browser extension APIs
- Computer use
- Excalidraw
- FastAPI
- MCP
- OpenAI
- React
- SQLAlchemy
- SQLite
- WebSockets
Built With
- fastapi
- javascript
- nextjs
- python
