Inspiration

AI can answer almost anything. So why is understanding still so hard?

You get stuck on an equation, an unfamiliar interface, a research paper or a bug. You open another tab, upload a screenshot, explain what you need, and get an answer in seconds.

Then a new problem starts: you have to work out how that answer connects to what you're looking at. Which part of the diagram is it talking about? Which button should you click? Where did that calculation come from?

AI understood the question, but it never became part of the experience.

A great teacher doesn't just hand over information. They look at the same problem as you, point at what matters, draw connections and guide you until it clicks. We wanted that on the computer.

What if your entire desktop became a learning environment? Not another chatbot, and not another search engine, but a contextual AI companion that makes understanding feel natural and immediate.

What it does

Quicky.AI is a context-aware desktop assistant. It floats above your apps and brings the explanation to your screen. It looks at your screen when you ask, reasons about your question, and guides you with speech, visual annotations and step-by-step instructions.

Traditional AI tells you the answer. Quicky.AI helps you see and understand it.

It understands what you're looking at. It uses the content in front of you, so you don't have to rebuild the context in another chat.
It explains visually. Pointers, arrows, highlights, geometric marks and work boards tie each explanation to the screen. It doesn't say "look at the element on the left". It points at it, and it builds the working beside your original problem.
It speaks while it teaches. Narration is coordinated with the visual steps, so you follow along instead of reading a wall of text.
It guides you through the process. For tasks where you need to act and see what changes, its instructions stay connected to the app and the task. The aim is to help you learn by doing, not do everything for you.

Three demos, one bigger vision

01. SOLVE | Mathematics. Ask about a geometry diagram. Quicky.AI identifies the elements, draws annotations and explains in organized steps, while the working builds up beside the problem. It's not just the right answer. It's why it's right.

02. THINK | Strategy through chess. Traditional AI suggests a move. Quicky.AI uses the board as context: it highlights pieces, draws arrows and explains threats directly on the board, so you follow the reasoning, not just memorize a move.

03. CREATE | Professional software. In a complex tool like DaVinci Resolve, a tutorial tells you which control to use, but you still have to find it. Quicky.AI points at the editing controls, timeline elements and interface components inside the application.

And this is only the beginning. A developer reading unfamiliar code, a researcher with a dense paper, a learner of a new language, an analyst with a spreadsheet, someone facing an unfamiliar online form. Different people, one shared problem: the information is right in front of us, yet understanding it means leaving the experience.

We want Quicky.AI to become a contextual intelligence layer for learning and navigating the digital world. It won't be limited to one subject, tied to one app, or a destination people have to visit.

Learning doesn't happen in one application. Neither should your tutor.

How we built it

Turning model intelligence into a natural desktop experience takes more than good answers. The system has to handle screen coordinates, structured visual instructions, windows, synchronized narration and a responsive interaction. We built a TypeScript monorepo with three core systems.

Desktop experience: Electron, React, TypeScript. A floating launcher offers Voice, Capture, Text and Canvas. A transparent overlay shows annotations and work without leaving your app. This meant handling native windows, capture permissions, display positioning and coordinate transforms.

Intelligence layer: Anthropic Claude, Node.js, Fastify. A local orchestrator applies privacy safeguards and sends the screen context to Claude, which returns a structured lesson. Each step has narration plus drawing commands such as [POINT] and [DRAW:arrow], and our app renders them over your workspace. Claude decides what to explain. Our system makes it visible.

Teaching engine: Cartesia. A teacher points while explaining, not seconds afterward. We use Cartesia for speech and built orchestration that streams each step and syncs the pointer and annotations with the narration.

Memory and semantic intelligence: Convex and Moss. Convex provides learner memory and session history as a foundation for personalization. Moss matches similar questions so earlier explanations can be reused instead of repeating the reasoning. These are modular with local fallbacks, and cloud features need their integrations configured.

Observability: Elastic, OpenTelemetry, Sentry. Optional tracing and monitoring help us find delays and failures. Typed interfaces, validation and automated tests keep the systems in step.

Privacy by design

Understanding your screen carries responsibility. Quicky.AI captures only when you ask, not by continuous background monitoring. It has privacy controls, sensitive-window handling and redaction safeguards, and it is designed not to keep raw screenshots as learner history. Intelligence should be accessible. Privacy should remain intentional.

Challenges we ran into

Syncing voice and visuals. In longer explanations our timing estimates drifted. In testing, drawings and narration were up to about nine seconds apart, and a pointer that lands after the explanation has moved on is confusing. We redesigned timing around the spoken steps. Timing isn't a design detail. It's part of understanding.
Drawing over real apps. Arrows overlapped diagram lines and work boards competed with content. We refined placement, offsets, contrast, pointer movement and layouts. The challenge was to draw the right thing, in the right place, at the right time.
Coordinating many moving parts. Capture, reasoning, streaming, narration and overlay rendering each have their own timing and failure modes. We worked through startup, configuration, cancellation, window positioning and cross-platform issues. A running backend doesn't mean the tutoring pipeline is ready.
Testing spatial understanding. Sounding correct isn't enough. We needed to know whether it points at the right place. On 13 synthetic screenshots with known targets, it identified the correct subject in all 13 cases and hit about 95% of pointing targets. This is a limited synthetic test, not a real-world accuracy claim, but it gave us a way to test spatial grounding.

Accomplishments that we're proud of

Turning AI answers into an experience built for understanding, combining screen understanding, visual reasoning, speech and annotations in one desktop workflow
Using one underlying approach across math, chess and creative software, without building a separate tutor for each
Structured drawing instructions that turn model output into visual guidance
Better sync between speech and annotations
Automated tests and evaluation tools for interaction state, rendering and coordinate accuracy

Our biggest accomplishment is how the interaction feels. You no longer read a chatbot reply and imagine how it maps to your problem. The explanation appears where the problem is.

What we learned

Intelligence is more useful when it is connected to context. A correct answer that shows where to look and what to do next is a different experience.
The best AI products pair probabilistic intelligence with carefully engineered systems. Claude provides the reasoning. Our app provides structure, timing, coordinates, rendering and user control.
Reducing friction can matter more than adding features. Technology should adapt to how people learn, not the reverse.

What's next

We envision contextual help as a natural part of computing: understanding a scientific diagram without switching tabs, learning a codebase while exploring it, and learning a new tool from guidance inside the workflow.

Next up: better conversational voice input, stronger visual grounding, more real-world evaluation, better cross-platform reliability and real learner personalization. We also want adaptive teaching: an assistant that knows when you need a hint instead of an answer, explains differently when the first try doesn't land, and builds on what you already know.

We're not building an AI that does your thinking. We're building one that helps you get better at it.

The internet made information accessible. AI made answers accessible. Quicky.AI is our step toward making understanding accessible, right inside the digital world.