AIdeaText, A Multimodal Reasoning Tracker

A multimodal educational application that tracks reasoning development through semantic graphs using Gemini.

Comment

Inspiration

"The real transformation happens when we stop grading what students produce and start assessing how they think."

What it does

AIdeaText was created to improve the writing, reading comprehension, and argumentation skills of students and teachers through what we call semantic reasoning graphs and Gemini's inference capabilities. It bridges the gap between written drafts and oral defenses, using Central Resonance Analysis (CRA) to visualize the "noise" and "signal" in a student's logic.

How we built it

  1. The application is a high-performance React-based SPA engineered with a focus on semantic data visualization.
  2. AI Engine: We utilized Gemini 3 Pro for deep "thinking-heavy" tasks like Central Resonance Analysis and Gemini 2.5 Flash for low-latency audio transcription and Socratic chat interactions.
  3. Semantic Mapping: We implemented a custom CRA pipeline. Instead of simple keyword extraction, the model clusters "semas" (minimal units of meaning) into Concepts, mapping their relationships as logical sequences, reinforcements, or contrasts.
  4. Data Visualization: Using D3.js, we built a force-directed graph system that renders these relationships in real-time. We used Recharts to provide a comparative "Radar" profile of the student's cohesion metrics across different media.
  5. Data Portability: To support academic research, we built a JSONL export/import system that allows for the creation of a persistent corpus of student reasoning development.

Challenges we ran into

  1. Schema Rigidity: Generating complex graph data (nodes and edges) in a single pass requires strict JSON adherence. We overcame this by leveraging Gemini's responseSchema to ensure the UI never breaks during parsing.
  2. Semantic Drift: One major challenge was ensuring the AI could detect when a student "switched topics" between writing and speaking. We implemented a Topic Verification layer in the alignment engine to penalize synthesis scores if the underlying concepts drifted too far apart.
  3. Audio Pipeline: Handling raw web audio blobs and converting them into high-fidelity transcripts via Gemini Flash required careful management of base64 encoding and MIME types to ensure accuracy.

Accomplishments that we're proud of

  1. The Triple-Graph System: Successfully visualizing and comparing three distinct reasoning sources (Written, Oral, and Socratic Interaction) in one unified dashboard.
  2. Synthesis Detection: Building an evaluator that can see if a student actually listened to the Socratic Tutor by checking if concepts from the chat appear in their later written revisions.
  3. Visual Logic Legend: Creating a "Visual Language" for reasoning—where straight lines represent logic, short dashes represent reinforcement, and long dashes represent contrast.

What we learned

  1. Thinking Budgets Matter: For complex tasks like CRA, providing a thinkingBudget significantly reduces hallucinations and results in more "conceptually dense" graphs.
  2. Multimodality is the Future of Assessment: We learned that students often express more complex evidence orally than they do in writing; AIdeaText makes this invisible growth visible to teachers.
  3. Structured Output is a Game Changer: Using Gemini's Type.OBJECT schemas allowed us to treat the AI as a reliable middleware that returns ready-to-render data structures.

What's next for AIdeaText

  1. Native Video Analysis: Using the Gemini Live API to analyze facial expressions and confidence levels during oral defense alongside the semantic transcript.
  2. Classroom Heatmaps: A teacher dashboard that aggregates all student graphs into a "Classroom Resonance Map" to identify common misconceptions in real-time.
  3. Historical Timeline: A "Reasoning Replay" feature that lets students see their semantic graph evolve over weeks of a single project.

Built With

  • d3.js
  • gemini2.5flash
  • geminipro3
  • jsonl
  • react
  • rechart
  • semas
Share this project:

Updates