Luminary

About the Project

Inspiration

One in 36 children in the U.S. is diagnosed with autism spectrum disorder, yet most classroom technology is built for neurotypical learners. We watched special education teachers spend hours manually rewriting worksheets—translating a generic fractions lesson into something that uses a student's favorite cartoon character, avoids their specific frustration triggers, and matches their preferred learning modality (visual, auditory, or reading). That manual process doesn't scale, and it means students often receive one-size-fits-all material that leads to frustration, disengagement, and behavioral escalation.

We asked: What if AI could do that adaptation instantly — and also detect when a student is struggling before the teacher even notices?

That question became Luminary.

What It Does

Luminary is an AI-powered special education platform with two connected interfaces:

For teachers:

Upload any worksheet (PDF, image, or text) and Luminary uses Google Gemini to automatically generate personalized versions for every student in the class — rewritten with each student's favorite characters, adjusted to their reading level, and avoiding their known frustration triggers.
A real-time dashboard shows each student's engagement percentage, frustration level (low / moderate / high), and session status at a glance.
Analytics and reports break down per-student performance, question-level difficulty, and AI-generated insights about where students are struggling.

For students:

A warm, character-themed learning interface where a student who loves Bluey sees fraction problems about Bluey sharing pizza slices, while a Minecraft fan gets block-themed questions.
Multiple learning modes: Visual (illustrated), Auditory (narration-ready), Reading (text-focused), and Kinesthetic ("hands-on") — automatically defaulting to each student's preferred style.
A behavioral frustration detection system that runs silently in the background, tracking rapid clicks, consecutive wrong answers, "I don't understand" button presses, and distress keywords in chat. When the frustration score crosses a threshold, the system automatically triggers an Auto-Reframe — Gemini generates a gentler, step-by-step breakdown of the same problem using the student's favorite character as a guide.

How We Built It

We built Luminary as a full-stack React application:

Frontend: React 18 with React Router v6 for client-side routing across 13 screens (landing, teacher auth, teacher dashboard, upload, student profiles, reports, student auth, student home, assignment selection, curated lesson, auto-reframe, and completion). We used Recharts for data visualization in the teacher reports view.
AI Layer: Google Gemini 2.0 Flash via the @google/generative-ai SDK, called directly from the client. We built three core AI functions:
- extractTextFromFile() — multimodal extraction from uploaded PDFs/images
- adaptLesson() — generates per-student adapted content with character themes, quiz questions, hints, and reframe explanations
- generateReframe() — creates step-by-step breakdowns when a student is struggling, using their favorite character to bridge the emotional gap
Frustration Detection: A custom React hook (useFrustration) that computes a real-time 0–100 frustration score from four behavioral signals: rapid clicking (3+ clicks in 8 seconds = +25 pts), consecutive wrong answers (3+ = +30 pts), "I don't know" button presses (2+ = +20 pts), and flagged distress language (+25 pts). When the score hits 70, the system auto-navigates to the reframe screen.
Backend: An Express.js server with Multer for file upload handling, plus a secondary Ollama/Gemma integration for local LLM inference as a fallback.
Auth & Data: Firebase Authentication for teacher/student login flows, Firestore for user profiles (with role-based access), class code management, and assignment submission tracking.
State Management: React Context (LessonContext) for storing AI-adapted lesson data across the app, with localStorage persistence so adapted lessons survive page refreshes.

Challenges We Faced

Structured JSON from LLMs: Getting Gemini to consistently return valid JSON in our exact schema was the biggest technical challenge. Even with explicit instructions to "return ONLY valid JSON," the model would sometimes wrap output in markdown code fences or deviate from the requested structure. We built a cleanup pipeline (stripping fences, catching parse errors, falling back to mock data) to handle this gracefully.
Frustration detection tuning: Calibrating the frustration score thresholds was more art than science. Set the threshold too low, and the auto-reframe fires on normal exploration. Set it too high, and a struggling student doesn't get help. We iterated on the weighting system (25/30/20/25 point distribution across four signals) and the rapid-click time window (8 seconds) through playtesting.
Personalization at scale: Adapting a single worksheet for multiple students means making sequential Gemini API calls — one per student. With 4 students, that's 4 separate generation calls, each taking a few seconds. We had to design the UI with progress indicators and handle partial failures gracefully (if one student's adaptation fails, the others still succeed).
Dual-interface UX: Designing two completely different experiences (teacher dashboard vs. student learning interface) that share the same data layer required careful routing and state management. The teacher needs data-dense analytics; the student needs a calm, minimal, character-themed environment. Balancing these in one codebase was a constant design challenge.

What We Learned

LLMs are remarkably good at pedagogical content adaptation when given rich student profiles — the character-themed rewrites felt genuinely warm and engaging.
Behavioral signals (click patterns, error streaks, language analysis) can approximate emotional state surprisingly well, even without biometric sensors.
Building for accessibility from the start (multiple learning modes, frustration-aware UX, sensory preference awareness) makes the product better for everyone, not just neurodivergent users.
Firebase + Gemini is a powerful combination for rapid prototyping — we went from zero to a working 13-screen app with real AI integration in a single hackathon sprint.