Hands Remember

Turn disappearing know-how into a living apprenticeship—without flattening the master's voice.

Comment

Hands Remember — Devpost submission

Status: release-ready; only the public video and Devpost project URL remain to be attached.

Title

Hands Remember

Tagline

Turn disappearing know-how into a living apprenticeship—without flattening the master's voice.

Category

Collaborative Partner

Elevator pitch

Hands Remember is a consent-aware multimodal apprenticeship partner. It turns scattered workshop evidence—voice, photographs, notes, exceptions, and corrections—into a source-linked living knowledge map. It finds ambiguity, asks the most valuable clarification, proposes an inspectable patch, and changes future teaching only after authorized human confirmation. A separate learner memory lets later sessions adapt without confusing one person's experience with canonical craft knowledge.

The proof is a state change, not a chat answer: one master's correction creates canonical version two, and a fresh apprentice session teaches the revised instruction while another learner's memory remains invisible.

Inspiration

Tacit craft knowledge rarely lives in a clean manual. A master may demonstrate a motion differently from how a note describes it, use local vocabulary with no direct translation, or recognize material readiness through sound, resistance, and timing. Transcription preserves words. Ordinary RAG retrieves passages. Neither makes contradiction useful, asks the right next question, or proves a correction changed what will be taught later.

Hands Remember treats knowledge as something people refine together, not something a model silently declares.

What it does

  1. Ingests consent-classified source bundles and preserves immutable source, access, hash, and time-offset provenance.
  2. Uses Gemini 3.7 Flash to extract candidate claims; extracted content never begins as truth.
  3. Maps confirmed, candidate, disputed, and superseded knowledge in a versioned claim graph.
  4. Finds the highest-value ambiguity and proposes a bounded, source-linked correction.
  5. Sends the immutable patch package to independent Gemini 2.5 Pro and Gemma 4 critics; both have can_commit=false.
  6. Commits or rejects only after the master explicitly decides, preserving history and rollback rules.
  7. Retrieves only confirmed canonical knowledge for the apprentice while keeping learner memory identity-scoped, inspectable, and deletable.
  8. Opens a fresh session and proves the revised claim changes the next lesson.
  9. Provides evidence-calibrated attempt feedback and abstains when observation confidence is insufficient.

How we built it

The backend is Python/FastAPI with Google ADK and Gemini 3.7 Flash on Vertex AI. Typed tools separate retrieval, clarification, patch drafting, review, coaching, and memory. The ADK run has aggregate timeout, LLM-call, tool-call, and draft-mutation budgets; it may create at most one pending patch and cannot commit it.

Five bounded model duties are exposed as receipts:

  • Gemini 3.7 Flash for ADK collaboration and structured extraction
  • Gemini 2.5 Pro for contradiction and provenance critique
  • Gemma 4 for an independent knowledge-patch risk signal
  • Gemini Embedding for confirmed knowledge and learner retrieval
  • multimodalembedding@001 for consent-gated source-to-step evidence
  • Gemini Live 2.5 Flash provides a ticketed realtime voice bridge

Canonical claims, source consent, learner memory, attempts, semantic vectors, model receipts, upload intents, and deletion outboxes have explicit schemas. Firestore snapshots are published as immutable generations behind a compare-and-swap manifest. Versioned Cloud Storage objects are created through a crash-safe upload saga and deleted through a retryable outbox that enumerates all generations.

The React/TypeScript studio places source evidence, graph state, uncertainty, patch diff, critic consensus, and the apprenticeship view in one editorial worktable.

Consent, knowledge, and memory safety

Callers cannot forge provenance: quotes and offsets are reconstructed server-side from immutable transcripts. Patch confirmation revalidates consent and access class inside the transaction. Withdrawing a source invalidates pending patches, scrubs every historical quote, transcript, learner citation, attempt citation, and derived vector, and schedules every Cloud Storage generation for deletion. Startup and periodic workers drain pending/failed media intents after crashes.

Canonical knowledge and learner memory never share an authorization path. The ADK and Live system context use confirmed claims only; learner-authored preferences are bounded and labeled untrusted. Production demo admission is required for every role, and the shared production reset is disabled.

Live release results

  • Promoted release: champion-20260819-v11
  • Cloud Run revision: hands-remember-00001-wel
  • Cloud Build: 61b24970-7b2d-4307-9989-fabc78f861c2
  • 46 backend tests passed; statement coverage: 83%
  • Ruff, Vitest, TypeScript, Vite, browser smoke, Playwright, Terraform, private candidate, and public runtime gates passed with zero application console errors
  • Google ADK used gemini-3.7-flash with bounded calls and confirmed-only retrieval
  • Gemini 2.5 Pro and Gemma 4 returned independent live reviews with can_commit=false
  • Gemini Live 2.5 Flash completed a real text/audio turn; its single-use ticket was rejected on reuse
  • Gemini text and multimodal embeddings returned live model receipts
  • Master confirmation produced canonical v2; a fresh learner session retrieved v2 and did not retrieve another learner's memory
  • Firestore conflict, process crash, upload ambiguity, consent withdrawal, versioned deletion, prompt injection, provider failure, and cross-learner isolation have dedicated regression coverage

Google Cloud architecture

  • Google Agent Development Kit
  • Gemini 3.7 Flash, Gemini 2.5 Pro, Gemini Live 2.5 Flash, Gemini Embedding, Multimodal Embedding, and Gemma 4
  • Vertex AI and Gemini Developer API
  • Google Cloud Run with one writer instance and concurrency eight for realtime sessions
  • Named Firestore database hands-remember
  • Versioned Cloud Storage evidence bucket
  • Cloud Build, Artifact Registry, IAM, Logging, and release-gated infrastructure

Data, consent, and honest limitations

The paper-cord judging corpus is an original synthetic product fixture, not cultural fieldwork or evidence of measured skill transfer. It is visibly disclosed and consent-safe. No real practitioner quote, personal media, or community knowledge is claimed. Live image embeddings do not persist image bytes. The current evaluation proves system boundaries and state continuity; it does not claim that a longitudinal apprenticeship study has already demonstrated improved human skill.

What we learned

The crucial design move was splitting “what the craft says” from “what this learner needs.” Once those stores were separate, consent, provenance, retrieval, and rollback became testable rather than aspirational. Multimodal fallback also had to rebase an entire request into one vector space; mixing live and fallback vectors silently drops evidence. Finally, review models are most trustworthy when they critique an immutable package and lack the tool that commits it.

Links

Testing instructions

Open the hosted application and enter the private judge admission code from Devpost's private testing-instructions field. Sign in as the master, inspect the source-linked ambiguous claim, choose 先看差異，不直接寫入, and verify that the patch remains pending while both critics report. Confirm the patch, wait for the canonical-version receipt, then switch to 學徒. The fresh session must display uses canonical v2 and the revised outside-mist / never-soak instruction. Production reset is deliberately disabled; the repository's ./scripts/verify.sh reproduces the isolated reset-and-confirm path locally.

Build disclosure

Hands Remember was newly built during the All Things Agentic Hackathon submission period. Libraries, Google services, AI coding assistance, synthetic corpus, generated concept art, and limitations are disclosed in the repository.

Built With

  • cloud-storage
  • fastapi
  • firestore
  • gemini-2.5-pro
  • gemini-3.7-flash
  • gemini-live-2.5-flash
  • gemma-4
  • google-agent-development-kit
  • google-cloud-run
  • react
  • vertex-ai
Share this project:

Updates