GemmaStudy

Multimodal Socratic study OS with PDF/code ingestion, KaTeX math, and active-recall flashcards across Cloud Gemma 4, an Exo USB4 distributed Mac mesh, and on-device Android.

Comment

Inspiration

During exam preparation, we observed major pain points in how students study with modern AI:

  1. The Homework Trap: Students paste problem sets into generic chatbots, receive direct solutions, and bypass the cognitive struggle required for genuine understanding.
  2. Extreme Tool Fragmentation: Students jump between five different browser tabs—one for lecture slides/PDFs, another for a chatbot, another for Anki flashcards, another for code debugging, and another for formula references.
  3. Connectivity Fragility: When campus Wi-Fi drops or during power grid outages, cloud-only AI stops functioning completely.

We set out to build GemmaStudy: an all-in-one multimodal Socratic learning operating system where students never have to leave their workspace, powered by Google DeepMind's open Gemma 4 models across cloud, distributed Mac mesh, and on-device mobile hardware.

What it does

  • Universal Socratic Dialogue: A pedagogical companion that guides students across any subject (Computer Science, Mathematics, Sciences, Humanities, and Languages) with step-by-step hints, analogies, and verification questions rather than spoiling answers.
  • KaTeX Step-by-Step Proofs: Renders multivariable calculus, linear algebra, and physics derivations with clean formatting (e.g., $\oint_C \mathbf{F} \cdot d\mathbf{r} = \iint_S (\nabla \times \mathbf{F}) \cdot d\mathbf{S}$).
  • Collapsible Reasoning Traces: Real-time streaming disclosure of Gemma 4's internal thoughts and reasoning process.
  • Multimodal Document & Code Ingestion: Ingests multi-page research PDFs (.pdf), source code files (.c, .cpp, .py, .java, etc.), and diagrams via one-click attachment with full text extraction and complexity analysis ($O(N)$).
  • Exo Distributed Mesh (USB4): Shards model weights across multiple Apple Silicon devices (MacBook Air M5 + Mac Mini M2) over a 40 Gbps USB4 bridge, pooling unified memory for fast local inference.
  • Android Companion App: Native on-device mobile companion with whiteboard camera OCR, perspective cropping, and customizable active-recall flashcard decks (5, 8, or 10 cards per deck).

How we built it

  • Desktop Workstation: Built with React, Vite, and KaTeX, featuring a minimalist dark-mode interface, real-time Server-Sent Events (SSE) token streaming, and an extensible Theme Engine.
  • Backend & Inference Gateway: Engineered a Node.js Express & WebSocket gateway that orchestrates a 4-tier hybrid cluster:
    1. Cloud Flagship: Google AI Studio Gemma 4 31B IT (256k context window).
    2. Exo Distributed Mesh: Gemma 4 sharded across MacBook Air M5 (24GB) and Mac Mini M2 (8GB) via 40 Gbps USB4 using Exo Labs MLX pipeline ring parallelism.
    3. Local LM Studio: Gemma 4 26B A4B QAT and E2B on local ports.
    4. Mobile Edge: Gemma 4 LiteRT running on-device on Android.
  • Document Pipeline: Integrated pdf-parse for multi-page document parsing and Gemini vision inline_data for multimodal image comprehension.
  • Android App: Developed in Kotlin with Jetpack Compose, featuring camera perspective transforms and flashcard generation.

Challenges we ran into

  • Pedagogical Guardrails vs Answer Leaks: Standard LLMs default to outputting direct solutions. We engineered a two-stage cognitive scaffolding system prompt that requires the model to explain underlying mechanics and provide hints before revealing proofs.
  • Reasoning Stream Parsing: Gemma 4 streams thought tokens intermixed with text tokens and JSON payloads. We implemented stream-splitting decoders in JavaScript and Kotlin that route thoughts to collapsible accordions while preserving pure JSON payloads for structured flashcards.
  • Distributed Inference over USB4: Configured peer-to-peer USB4 / Thunderbolt networking (bridge0) to shard large LLM layers across disparate Apple Silicon chips without bandwidth bottlenecks.

Accomplishments that we're proud of

  • Full 4-Tier Hybrid Cluster: Seamlessly unified Google AI Studio 31B, an Exo USB4 distributed Mac mesh, and on-device Android execution.
  • Fluid Socratic Interface: Sub-20ms streaming token UX with KaTeX proofs, thinking trace disclosure, and PDF/code attachments.
  • Complete Cross-Platform Workflow: Deep desktop study workstation synchronized with an on-device Android mobile companion.

What we learned

  • Prompt Engineering as Curriculum Design: Guiding a model to teach Socratically requires treating system prompts like instructional pedagogy.
  • Edge & Distributed Inference is Ready: Quantized Gemma models running locally across USB4 rings and mobile NPUs deliver responsive, private token streaming with zero recurring cloud costs.

What's next for GemmaStudy

  • Canvas LMS Integration: Automatic synthesis of active-recall decks from university course announcements and lecture repositories.
  • Dorm Mesh Study Rooms: Peer-to-peer local Wi-Fi collaborative study sessions without cellular data dependence.

Built With

Share this project:

Updates

Submission history