Nityam

Nityam is a personalized, collaborative AI tutor that listens, remembers every session, adapts how it teaches, and won't move on until a concept truly clicks

Comment

Inspiration

Hours into a lecture, hours into a YouTube video, and the concept still would not click. That was the moment this project actually started: not frustration with a lack of content, there has never been more free content in the world, but frustration with what that content asks of you. A recorded lecture cannot tell when you have stopped following. A textbook cannot ask you a question and wait for your answer.

The newer tools did not fix this either. ChatGPT and NotebookLM are extraordinary at processing information, but they are prompting tools, not teaching tools. They wait for you to already know what to ask, and the moment you stop prompting, they stop helping. Real learning is not information retrieval. It is a relationship: someone who notices you are lost before you say so, who adapts the explanation to how you actually think, and who checks that the idea landed before moving on.

That is what we wanted to build. Something closer to sitting across from a real teacher, one who never gets tired, who ties a hard physics concept to whatever you actually care about, whether that is cricket or circuit design, who quizzes you the moment an idea should have landed, and who genuinely will not let a session end until the topic is actually understood, not just covered.

We named it Nityam, meaning persistent, or eternal. A tutor whose whole purpose is to still be there for you, session after session, remembering exactly where you left off.

What it does

Nityam is a live, voice based tutor that behaves less like a chatbot and more like an actual teacher. You talk to it the way you would talk to a person, and it teaches back the way a good one does: it draws out what you already know before adding to it, checks your understanding with a real question instead of assuming an explanation landed, and only re-explains, differently, when a gap turns out to be real.

Every lesson is grounded in real material. You can upload a YouTube lecture, and because we have already grounded a full NCERT textbook chapter by chapter, the tutor can immediately connect what that video taught to what your syllabus actually covers. Upload your own lecture recordings or slides, and it reads the handwriting on the board and the diagrams on the slide, not just the transcript, so the images themselves become something you can ask about and it can bring back to you exactly when they matter.

Across one conversation, the tutor:

  • Writes explanations and formulas on a shared board as it teaches
  • Pulls up the exact textbook figure a concept needs
  • Runs a quiz checkpoint the moment an idea should have landed
  • Builds an interactive simulation when a concept is better discovered than described

All of this happens inside the same natural conversation, without ever feeling like four separate tools.

What makes it a collaborative partner rather than a smarter answer key is memory. After every session, the tutor updates what it knows about you: what you are weak at, what teaching style has actually worked, what you were still confused about when the call ended. That memory is grounded in real Socratic teaching technique and in what actually works pedagogically, so the tutor is not only learning who you are, it is applying real teaching practice and a working understanding of student psychology to how it teaches you, and it gets better at both with every session.

How we built it

Nityam is built on Google's Agent Development Kit, orchestrating five agents behind a single Gemini Live voice connection. A VoiceAgent handles the actual conversation, hearing and speaking through Gemini's real time bidirectional streaming, and routes to four specialist agents as the lesson calls for them:

  • One that writes on the shared board
  • One that finds the right textbook page
  • One that runs quiz checkpoints
  • One that builds interactive simulations

These specialists run as their own asynchronous processes so the tutor never goes silent waiting on one of them. While a simulation is being generated in the background, the voice agent keeps teaching, and the moment the specialist finishes, its result is woven back into the conversation at a natural pause rather than interrupting mid sentence.

Underneath the agents sits SMRITI, a three tier memory layer built specifically so the tutor's personalization is real rather than a prompt trick:

  1. Workflow memory (held in Redis) — what is happening in the conversation right now: the live turn buffer that every agent reads from and writes to during a session.
  2. Episodic memory (written once when a session closes) — the permanent record of what actually happened in that session, turn by turn.
  3. Long term memory (carries across every session) — a dynamic personal memory profile of the student's mastery and interests, and a separate teaching memory of what pedagogical approach has actually worked for them.

Every claim written into long term memory cites the exact session and turn it came from, so the tutor's memory can always be traced back to real evidence rather than an unverifiable guess.

Feeding that memory is Shruti, our own pipeline for turning a recorded lecture into grounded teaching material. Shruti does not just transcribe a video. It reads the handwriting recovered from the board, resolves the deixis in what a teacher says out loud, such as "this term" or "here", back to the actual thing being pointed at, and synchronizes the beats of what was said with what was written, so the resulting knowledge graph is not just text but a citable, timestamped record of exactly how a concept was actually taught in class. That grounded material is what the tutor draws from instead of generic textbook content.

The whole system runs on Google Cloud:

  • The backend, all five agents, and the memory layer are deployed as a single Cloud Run service with continuous deployment through Cloud Build, so every change ships automatically.
  • Firestore holds episodic and long term memory along with grounding data.
  • Memorystore for Redis holds workflow memory and the live event stream.
  • Cloud Storage holds every generated artifact durably so a reload never loses a student's simulation.

We also built our own observability layer, the Smriti Observatory, which watches every memory read and write live as a session runs, including the writes our own validation rules reject, because a memory layer where every write always succeeds is indistinguishable from one with no rules at all.

Challenges we ran into

The hardest problem was keeping the voice conversation, the board, and generated artifacts in sync without ever making the student wait in silence. A simulation can take up to thirty seconds to generate, and a real teacher does not just stop talking for thirty seconds. We had to build a mechanism where the voice agent keeps teaching naturally while a specialist works in the background, then weaves the result back in at a natural pause rather than announcing that something is loading. Getting that pacing right, so it never talks over the student and never goes quiet for too long, took real trial and error against the real timing of a live model.

A closely related challenge was balancing grounding against latency. It would be easy to make responses instant by letting the model improvise, but an improvised physics explanation is worse than a slow correct one. We had to make sure every explanation stays grounded in the actual textbook and the actual lecture the student is studying from, without the retrieval and validation work behind that becoming something the student can feel as dead air.

The other hard problem was memory itself. A tutoring conversation can run long, and it was easy for the model to lose track of what had actually been taught earlier in the same session, or for the conversation to drift away from the concept it was supposed to be teaching. We solved this by never writing long term memory mid conversation, only once at the end of a session through a single validated pass, so a claim about a student is only ever written when there is enough evidence to support it, and by keeping a live workflow memory of the current session that every agent can check before acting, so the tutor stays anchored to what has actually happened rather than drifting.

Accomplishments that we're proud of

The thing we are most proud of is that we would genuinely use this ourselves to study. Compared to a text based tool, Nityam is an actual learning environment: you can bring your own textbook, select an image out of it, ask a question about that image, and get taught with the real diagram in front of you rather than a description of one.

The deepest technical accomplishment is the personal memory substrate itself: one shared memory layer that every agent reads from and writes to, across every mode and every conversation a student has. Because that memory is centralized rather than scattered across separate agents, the grounding stays consistent no matter which specialist is teaching at a given moment, and the tutor's understanding of a student never fragments.

Achieving real time teaching that stays grounded in what was actually covered in class, combined with a genuinely seamless multimodal experience across voice, board, textbook, and simulation, while keeping one shared memory underneath all of it, is the achievement we are proudest of.

What we learned

We learned how much of building a good agent is actually memory management, not just during a live conversation but in the quiet period right after it ends, when a session's transcript has to become a verified update to what the tutor knows about a student. We learned that memory can drift in ways that are easy to miss, and that you cannot trust an agent's memory layer unless you can actually see it working, which is exactly why we built the Smriti Observatory, a live view into every memory read and write as a session runs. Watching that data live, including the writes our own validation rules rejected, taught us more about our own system's real behavior than any amount of manual testing did.

We also came to appreciate just how much the Google ecosystem lowers the barrier to building an agent system like this. The Agent Development Kit's model for tools, sub agents, and streaming responses meant we could focus on the actual pedagogy instead of reinventing orchestration, and pairing it with Cloud Run, Firestore, and Memorystore gave us a genuinely low latency, production grade backend without having to hand build infrastructure most teams would spend weeks on.

On the engineering side, we learned how much modular design matters once a system has this many moving parts. Well defined tools, well defined agent boundaries, and clean interfaces between them were the difference between a system we could actually debug and one that would have collapsed under its own complexity.

Finally, we learned that user experience in a voice tutor is really a psychology problem as much as an engineering one. Perceived latency and actual latency are not the same thing, and a student's anxiety about silence matters just as much as how fast the backend actually is. Building an agent that feels patient and present, not just technically responsive, turned out to depend as much on understanding how people actually learn as it did on the engineering underneath it.

What's next for Nityam

There is a lot ahead of us:

  • We want the artifacts themselves to grow richer and more personalized, generating a wider range of interactive experiences rather than a fixed set of simulation types.
  • We want to support collaborative sessions where two or three students can learn together in the same conversation rather than only one on one.
  • We also want the underlying pipeline to keep improving on its own. As more lectures and sessions get processed, it should get better at recognizing what actually makes a good explanation versus a weak one, refining how it teaches based on real evidence of what works rather than a fixed set of rules.
  • Longer term, we want Nityam to work the way learning actually happens outside a desk: syncing with a phone so a student can photograph a question they are stuck on mid homework and start a session from exactly that point, with modes built for revision as well as active problem solving.

The larger goal is for Nityam to become the best teacher available to any student, regardless of where they are, in their own language and dialect, adapting to however they learn best. Good teaching should not depend on which school a student happens to be in. We want to build toward a tutor that gives every student, everywhere, that same personal, patient attention.

Built With

Share this project:

Updates

Submission history