Senti

Teacher-student interactivity from home, Senti will plan, generate and walk you through any academic subject you ask for all while seeing your whiteboard live.

Comment 1

Inspiration

The idea came to me a while back, but I wrote it down until the day came and this hackathon nudged me to build it. Seeing us students solve on a whiteboard or pdf and discussing the solutions felt like there was a gap to be filled.

What it does

Senti fills this gap by being there, seeing your step-by-step solutions and correcting you while explaining. Senti can also start topics from scratch, generate learning plans and store them for you to come back to. Getting to experience an okay-decent level of interactive education was never really a solution a found, youtube videos or blogs being the best thing to learn from. Now you get a live tutor, that acts as a human and with the entire web knowledge ready to be delivered within a chat.

How we built it

Senti uses the Gemini Live api, with ephemeral tokens so that the client is directly connected through a websocket to the live service. I then built a suite of tools for the agent to use, and distilled the teaching experience with clear system instructions. Senti can write equations on the board, draw graphs, generate learning plans by sending it off to a better model - gemini 3 flash preview - and then view the current learning plan as well as save and retrieve learning plans.

Challenges we ran into

Showing a constant live feed to the agent was a tricky one. The current implementation revolves around a debounce from the user's last pen touch, an extra cautionary stale timer regardless of user input and converting the whiteboard into an image to be sent to the agent. The biggest problem is the performance, as you can feel a noticeable stutter when it's time to capture the board. I found out about tldraw's (the library used for the whiteboard) agent starter kit, which could've helped but ran out of time to really make use of it.

Accomplishments that we're proud of

I'm extremely proud of the seamless integration between the whiteboard and the agent. It really feels like the agent is there with you, in control when it needs to, and the ability to ask it anytime about the last thing you wrote down feels like magic every time.

What we learned

I'm really just realizing the potential for this use case of live agents. I've discovered that human-level interactions with a dedicated purpose - like teaching - is becoming more and more seamless. I'm excited to see how this can be implemented in other areas, like teaching programming, as well as totally different domains, say like gaming.

What's next for Senti

Senti will probably be mostly used by school students, thus the plan is clear: give Senti the best abilities for it to fully guide you through your personalize problems. This means 1) analyzing homework papers and solutions to start a structured solving session 2) adjust according to different curriculum 3) generate better images 4) offer a better notebook-like experience that can be exported by the end of the session.

Built With

Share this project:

Updates