UnderStudy

Project Story

Text Description

Understudy is an adaptive Socratic onboarding partner that diagnoses exact developer knowledge gaps and turns team friction into better documentation.

New hires spend weeks re-discovering the same gaps that the person before them already struggled through. Documentation goes stale because there's no feedback loop — the org only learns what's confusing when the same question appears in Slack for the third time. Existing onboarding tools dump massive doc libraries on new developers and hope for the best.

Understudy flips this: instead of broadcasting manuals, it asks diagnostic questions first, identifies each developer's exact misconceptions, explains only the specific gap grounded in real repository documentation, and feeds anonymized friction signals back into an org-wide Documentation Debt report. Every new hire makes the next one faster.

The system runs two layers in a flywheel. The Personal Layer maintains a private per-user Skill Graph tracking confidence across concepts — returning sessions open on weak spots first. The Org Layer watches for patterns: when multiple independent new hires stumble on the same concept, it raises an anonymized Documentation Debt signal to the engineering team, complete with actionable remediation tasks like "Add a worked example for the expiration time claim in onboarding docs." No raw transcript ever leaves the personal layer.

Four decoupled agents — Orchestrator, Socratic Dialogue, RAG/Context, and Reflection — follow a strict single state-owner rule: only the Orchestrator ever writes to storage, making the org-level aggregate trustworthy and auditable in one file.

Features and Functionality

• Socratic Diagnostic Questioning — Gemini 3.5 Flash-Lite asks open-ended diagnostic questions before explaining anything, using Pydantic structured outputs for type-safe, validated JSON responses directly from the model.
• Grounded Source Citations — Every explanation cites the specific doc or code chunk it came from. Clickable inline citation badges open a live source-text inspection modal so the user can verify the grounding.
• Per-User Skill Graph (Radar Chart) — A canvas-rendered spider chart maps confidence across concepts, updating in real time as the session progresses.
• Session History & Persistence — Full session transcripts, turn counts, and timestamps are persisted to Firestore and browsable from the dashboard.
• Multi-Concept Topic Progression — Sessions advance through concepts with adaptive ordering based on the user's weakest areas.
• Interactive Architecture Visualizer — A collapsible in-app diagram shows the 4-agent workflow and tech stack, so evaluators can understand the system without leaving the UI.
• Org-Wide Documentation Debt Flywheel — Automatically aggregates anonymized friction signals across all users and generates actionable remediation tasks (e.g., "Add a worked example for 'expiration time claim' in the onboarding docs").
• Offline Dev Mode — A StubDialogueLLM allows full end-to-end testing of the orchestration loop, skill graph, and doc debt pipeline without any API keys.

Technologies Used

Other Data Sources Used

• PyJWT — A small, real, well-documented open-source Python JWT library used as the demo onboarding corpus. The ingestion pipeline processes both documentation (docs/*.rst) and source code into 87 retrievable chunks, each tagged with a deterministic concept_id (e.g., usage.expiration-time-claim-exp). Concept IDs are assigned at ingestion time with no LLM call, so personal and org layers always join on the same key across re-ingestions.

Findings and Learnings

1. The most consequential bug was semantic, not a crash. Defining "low confidence" off the user's post-explanation state would have made the Dialogue Agent erase its own Documentation Debt signal — the exact feature that differentiates Understudy from a single-player tutor. The flywheel would have silently broken. Fixed by tying the org signal to in-session friction instead.

2. Retrieval quality bugs are invisible until you test real user phrasing. Plain TF-IDF missed an "expires" (query) → "Expiration" (doc heading) match with no stemming. Added a Porter-stemmed, digit-aware custom tokenizer — this also fixed RS256/HS256 being silently collapsed into the same token.

3. Single state-owner architecture pays off. Decoupling state ownership to one agent (verified by grep across the entire codebase, not just documented) made the org-aggregate trustworthy to reason about. When you can prove by code search that only one file touches storage, privacy guarantees become auditable, not aspirational.

4. Pydantic structured outputs eliminate an entire class of bugs. Using response_schema=SocraticTurnOutput guarantees type-safe JSON from Gemini every call — no regex parsing, no retry loops, no hallucinated field names. This moved LLM integration from "fragile glue code" to "typed contract."

5. Scoping is not faking. We deliberately chose a small, real corpus (PyJWT) instead of a large synthetic one. TF-IDF without semantic embeddings sometimes ranks a topically-adjacent chunk above the exact right one — that's exactly the gap Vertex AI Search would close in production. Documenting the limitation honestly is more valuable than hiding it behind a bigger corpus.

Built With

• cloud-run
• css
• fastapi
• firestore
• gemini
• google-adk
• google-genai
• html
• javascript
• pydantic
• python