Lumina Learn

A local-first AI desktop learning assistant with real-time screen vision, OCR, PyTorch activity recognition, vector memory, and RAG tutoring.

Inspiration

Most productivity and "focus" tools only tell you that you got distracted — a screen-time report at the end of the day, long after the moment you could have actually done something about it. We wanted a study companion that understands what you're doing right now, on your own screen, and gently nudges you back on track while you're still in the middle of the work — not a retrospective guilt trip.

We were also frustrated by how many "AI study assistants" require sending a continuous video feed of your screen to the cloud. Screens contain passwords, private messages, unfinished drafts, and other sensitive material. We wanted the opposite default: a tool that keeps raw frames on-device, reasons about them locally, and only reaches out to the network when the user explicitly allows it. That local-first constraint became the seed of the whole architecture.

What it does

Lumina Learn watches your screen (only when you turn capture on), and in near real time:

Captures and compresses frames on the desktop client and streams them to a local backend over WebSockets.
Runs OCR to extract on-screen text and context.
Classifies what kind of activity is happening (coding, reading, watching video, etc.) with a PyTorch CNN, alongside an educational-context classifier and a distraction score.
Stores session and embedding memory so it can retrieve relevant past context and tutor with retrieval-augmented generation (RAG) rather than answering blind.
Surfaces all of this — live OCR, activity confidence, suggestions, and analytics — in a React/Tailwind overlay shipped as a Tauri desktop app.

How we built it

We structured the project as a modular monorepo rather than one big app, since vision, OCR, classification, memory, and tutoring are genuinely separate concerns that needed to evolve independently:

apps/backend — a FastAPI service that owns WebSocket transport, persistence, and orchestration of the processing pipeline.
apps/desktop — a Tauri + React + TypeScript shell that captures the screen locally and renders the live overlay.
packages/vision — screen capture and OpenCV preprocessing before anything touches the model.
packages/activity-classifier — the CNN that classifies activity, plus training scripts and dataset tooling.
packages/memory — vector memory and RAG retrieval (Chroma as the primary store, FAISS available for embedded local indexes).
packages/ai-core — embeddings and pluggable reasoning providers, so local transformer models and future cloud backends share one interface.
packages/shared — Pydantic schemas and event contracts used across every service, so the WebSocket protocol has one source of truth.
infrastructure — Docker and Kubernetes manifests for shipping the backend as a real deployable service, not just a local script.

The pipeline itself is intentionally asynchronous end to end: frames are captured on the desktop, streamed over WebSockets, and processed through OpenCV preprocessing → OCR → CNN classification → context classification → distraction scoring → embedding memory, all before results flow back to the overlay. Sessions, frame events, and analytics persist through an async repository layer on SQLite in development, written against interfaces that don't assume SQLite so a move to PostgreSQL is a swap, not a rewrite.

For training, we built a seed screenshot generator and a one-command training script (train_activity_model.ps1) so the classifier isn't stuck depending on one hand-labeled dataset — anyone can capture their own screenshots with capture_training_screenshot.ps1, label them by activity and educational context, and retrain.

What we learned

Local-first has real architectural cost, and it's worth it. Keeping inference pluggable (local transformer models today, cloud backends later) while still defaulting to on-device processing forced cleaner boundaries between transport, storage, and inference than we would have written otherwise.
A heuristic fallback matters more than a fancy model. Development runs without a trained classifier checkpoint at all, using a deterministic heuristic classifier, which meant the rest of the system could be built and tested before the model was any good.
Redaction has to happen before storage, not after. Screen OCR will eventually capture something sensitive. We learned to treat the memory layer as untrusted-until-redacted and strip common secret patterns before anything gets embedded into vector memory.
Schemas as contracts save a monorepo. Centralizing event and data contracts in packages/shared was the single biggest thing that kept the backend, desktop app, and ML packages from drifting apart as they were built in parallel.

Challenges we ran into

Streaming compressed frames over WebSockets without stalling the UI. Balancing frame rate, compression, and MAX_FRAME_BYTES limits so the overlay stays responsive while the backend isn't overwhelmed was an ongoing tuning problem.
Making activity classification useful before we had good training data. We had to design the system so it was usable and testable with a heuristic classifier from day one, then swap in the trained PyTorch model without touching anything downstream.
Privacy without breaking usefulness. Redacting secrets out of OCR text before it reaches vector memory, restricting CORS and WebSocket origins, and gating operational APIs behind API_KEY all needed to happen without making the tool annoying or fragile to use.
Cross-stack packaging. Getting a Python/FastAPI backend, a Rust/Tauri desktop shell, and a React/TypeScript UI to build and ship together — including native packaging via pnpm build:desktop — took real trial and error across pyproject.toml, pnpm-workspace.yaml, and the Tauri toolchain.

What's next

Training the activity classifier on a larger, more diverse real-world screenshot dataset.
Expanding RAG tutoring so suggestions get more specific as vector memory accumulates more of a user's session history.
Moving the default persistence layer from SQLite to PostgreSQL for multi-session, multi-device use.
GPU-backed inference workers for the vision and classification pipeline in the Kubernetes deployment.

Built with: Python, FastAPI, WebSockets, PyTorch, OpenCV, OCR, Chroma/FAISS, React, TypeScript, Tailwind CSS, Tauri, SQLite, Docker, Kubernetes.

Repo: github.com/ibnshafi/lumina_learn