Noteslyt

Inspiration

Students study from scattered lecture videos, textbooks, and slides, but general AI chatbots answer from thin air — not tied to course material or to what the student already knows. We wanted a companion that only claims what it can cite: every answer opens the exact page, slide, or timestamp it came from, and clearly refuses what the material doesn't cover.

That became NotesLyt: one project = one private knowledge base, with a learner model that tracks what you actually master.

What we learned

Grounding is a systems problem, not just prompting. Citations must be stripped server-side unless the locator was really retrieved, or the UI will happily show fake links.
Chunkers must never cross a locator boundary (page / slide / segment / sheet), or citations break.
Personalization needs both memory and math: Bayesian Knowledge Tracing for per-topic mastery + IRT-style ability for overall level + de-duplicated question selection so practice stays novel.
Evaluation keeps you honest: RAGAS-style faithfulness / relevancy / context precision-recall + refusal accuracy, plus seeded simulated students to prove mastery gains and low repetition.

Learner update (BKT):

$$P(L_t \mid obs) = \frac{P(L_{t-1}) \cdot (1-P(S))}{P(L_{t-1}) \cdot (1-P(S)) + (1-P(L_{t-1})) \cdot P(G)}$$

for a correct answer, with slip $P(S)$, guess $P(G)$, then $P(L_t) = P(L_t \mid obs) + (1-P(L_t \mid obs)) \cdot P(T)$ with transition $P(T)$.

Ability (Rasch / 1PL IRT):

$$P(correct \mid \theta, b_i) = \frac{1}{1 + e^{-(\theta - b_i)}}$$

How we built it

Stack: FastAPI + Vite/React served on the same port. Postgres + pgvector for hosted, SQLite + ChromaDB for zero-install desktop. Same API, two backends.

Multimodal ingestion: app/ingestion/registry.py dispatches by format id (pdf, pptx, xlsx, audio, video…). Every file becomes SourceUnits with locators. PDFs additionally render each page to PNG and embed as image-kind chunks so diagrams/figures are retrievable. Audio/video go through transcription, then transcript vectors.
Grounding: retrieval always takes project_id — no search-everything. match_chunks is project-scoped (tenancy guarantee). Ownership checks return 404, never 403, so IDs aren't enumerable.
Tutor chat: cited excerpts, click-to-open PDF page / slide / timestamp in SourceReader.
Adaptive assessment: Studio generates MCQ / short-answer / numerical tagged by topic + source location + difficulty, verifies answer keys, avoids repeats, gives cited feedback + weak-topic / misconception report.
Learner model: BKT per-topic bars (MasteryBar), Learning Profile page with IRT ability, diagnostic quiz for cold-start.
Evaluation: eval/run_ragas.py + eval/simulate_students.py on team-built Web Tech testset (known source locations + off-material queries).

Challenges faced

Embedding width drift: swapping models (1536-d OpenAI → 768-d local) broke the chunks.embedding column. Fixed with migration-time width check, degraded boot + 409 guards with re-embed hints, and shared app/reembed.py core.
Waking the embedder evicted transcription: probing embeddings killed single-resident local engines. Fix: GET /system/settings/status never probes unless ?probe=true.
Fake citations: solved by server-side citation stripping and stable uuid5(source_id, index) chunk ids so re-ingest upserts instead of duplicating.
Windows + PowerShell quirks, LibreOffice for slides, FFmpeg bundling for desktop — all handled in scripts, not docs.