architecture diagram
UI

SightLine Tutor

SightLine Tutor is a real-time multimodal AI tutor that can see a student's homework through the camera and explain the solution out loud. Students can ask questions verbally and interrupt the tutor naturally, creating a more conversational learning experience than traditional text-based chatbots.

Inspiration

Many students solve homework on paper, but most AI tutoring tools require them to type equations or upload screenshots. This creates friction and interrupts the learning flow.

I wanted to explore how multimodal AI agents could make learning feel more natural: students should be able to simply show their work and talk to the tutor, just like they would with a real teacher.

The Gemini Live API made it possible to build a real-time tutoring experience that combines vision, voice, and conversational interaction.

How It Works

SightLine Tutor allows students to:

Show a homework problem through the camera
Ask questions using voice
Interrupt the tutor for clarification

For example, if a student shows the equation:

$$ 5(2x - 3) + 4 = 19 $$

The tutor explains the steps:

$$ 5(2x - 3) + 4 = 19 $$

$$ 10x - 15 + 4 = 19 $$

$$ 10x - 11 = 19 $$

$$ 10x = 30 $$

$$ x = 3 $$

Students can interrupt at any point, for example:

"Why did you multiply the 5 with both numbers?"

This creates a more natural and interactive tutoring experience.

Architecture

SightLine Tutor uses a simple real-time architecture:

Frontend

Next.js
Camera preview
Microphone input
Audio playback

Backend

FastAPI
WebSocket streaming
Gemini Live session management

AI Model

Gemini multimodal model via Gemini Live API

Cloud Infrastructure

Backend & Frontend deployed on Google Cloud Run

Interaction Flow

Student provides camera image and voice input
Next.js frontend captures the inputs
Inputs are streamed via WebSocket
FastAPI backend (Cloud Run) manages the live session
The backend communicates with Gemini Live API
Gemini generates a real-time spoken tutoring response

Challenges

Real-time streaming
Coordinating camera input, voice interaction, and AI responses in real time required careful handling of WebSocket communication.

Vision reliability
Ensuring the camera captured readable equations was important for consistent model understanding.

Interruptible conversations
Allowing the student to interrupt the tutor mid-explanation required maintaining an active live session with Gemini.

What I Learned

This project showed how powerful multimodal AI agents can be for education.

Key takeaways:

Vision + voice creates a much more natural interface than text alone.
Real-time interaction makes AI feel like a collaborator rather than a tool.
Multimodal models enable new learning experiences that were difficult to build before.