SAGE - VLM Student Workspace

Inspiration

As a student, one of the biggest challenges I face daily is keeping track of everything—from looming assignments and quizzes to monitoring dietary macros and workout targets. While I interact with AI models every day, standard LLMs lack ambient context and feel disconnected from my actual workflow, rather than acting as a true personal assistant.

This realization drove me to create SAGE (Student Assistant for Growth and Education)—an ambient "second mind" designed to keep student tasks, schedules, health metrics, and study notes organized in an automated way. To make a real-time visual sidecar possible without lagging the user's workspace, SAGE relies on ultra-fast multimodal inference powered by Nebius AI Studio and NVIDIA GPU acceleration.

What it does

SAGE acts as a unified academic and personal sidecar:

• Multimodal VLM Screen Analysis: Captures live screen/tab context via Nebius AI Vision-Language Models to extract assignment deadlines and course details automatically without manual data entry.
• Automated Schedule & Push Notifications: Automatically populates course schedules and fires native browser push notifications prior to upcoming deadlines.
• Schema-Driven Dynamic UI Widgets: Generates context-aware tracking cards (such as workout progression, test score comparisons, and study counters) on the fly based on natural language requests.
• Macro & Health Tracking: Tracks daily caloric and macronutrient targets with real-time progress visualization.
• RAG Document Search: Indexes study notes and course documents for instant context-aware Q&A using accelerated vector embeddings.

How we built it

SAGE is built on a high-performance web-first architecture optimized for low-latency AI inference:

• Inference & Acceleration: Powered by Nebius AI Studio for high-throughput LLM function calling and VLM vision processing, leveraging NVIDIA Tensor Core GPUs to achieve near-instant response times for live screen analysis and chat generation.
• Frontend: Built with Next.js, React, and Tailwind CSS, utilizing a modular Widget Registry Pattern to map structured backend schemas directly to dark-mode UI components.
• Backend: A fast FastAPI microservice orchestrating tool routing, state persistence, and vector indexing.
• Local State & RAG: Uses lightweight JSON schema contracts (widgets.json, schedule.json, macros.json) for zero-latency local state sync alongside fast RAG document indexing.

Challenges we faced

1. High-Throughput Multimodal Processing

Running continuous visual context extraction required optimizing frame transmission over API endpoints. By utilizing Nebius AI's high-speed inference pipeline backed by NVIDIA hardware, we minimized first-token latency. We optimized frame downsampling and canvas JPEG compression ratio ( q ), bound by:

$$ \text{Payload Size} = \iint_{0}^{W, H} f(x, y) \, dx dy \quad \text{where} \quad \text{Resolution} \le 1280\text{px} $$

This ensures visual payloads process in milliseconds while maintaining a smooth ( 60 \text{ FPS} ) UI rendering experience.

2. Deterministic AI UI Generation

Allowing an AI agent to generate arbitrary UI components often risks build breaks or hydration errors. We solved this by implementing a strict JSON schema contract. Instead of generating raw React code, SAGE outputs structured JSON payloads that our frontend registry renders deterministically.

3. VLM Date & State Extraction

Fine-tuning prompt structures so the VLM accurately extracts relative dates (e.g., "Due next Friday at 11:59 PM") and converts them into standardized ISO timestamps required rigorous tool-loop handling and error fallback logic.

What we learned

• Inference Speed Matters: Leveraging Nebius AI and NVIDIA GPU acceleration transforms AI from a slow chat interface into an instantaneous, ambient co-pilot that feels like part of the operating system.
• Schema over Code Generation: Decoupling AI data intent from React rendering via JSON schema contracts yields rock-solid UI stability with zero runtime crashes.
• Multimodal Efficiency: Combining visual context extraction with structured tools makes conversational agents feel truly personal and context-aware.

What's next for SAGE - VLM Student Workspace

• On-Device & WebGPU Optimization: Exploring hybrid inference setups combining Nebius AI Cloud for heavy reasoning tasks with local WebGPU / NVIDIA TensorRT-LLM edge acceleration for instant offline screen processing.
• Direct LMS Integration (Canvas / Moodle / Blackboard): Moving beyond visual screen reading by adding two-way API webhooks to automatically sync course syllabi, assignment grades, and submission portals in real time.
• Cross-Device Mobile Companion App: Extending browser push notifications to native iOS/Android lock-screen alerts via a lightweight PWA/React Native companion app.
• Peer Collaboration & Shared Study Spaces: Allowing students to share localized RAG indexes (lecture notes, exam prep sheets) and custom tracking widgets across study groups.

Built With

• browser-notifications
• fastapi
• json-schema
• llm
• nebius-ai
• next.js
• nvidia
• ocr
• python
• rag
• react
• render
• tailwind-css
• typescript
• vercel
• vision-language-model
• vlm
• web-apps