Project- Arivu

Today, we consume massive amounts of dense information—research papers, corporate documents, textbooks—but our methods are painfully passive. Standard text-to-speech readers are robotic and emotionally flat. Furthermore, for neurodivergent individuals, losing focus or failing to grasp a singular concept can derail an entire study session. If you don't understand something, you have to break your workflow, stop the audio, run a web search, and entirely lose your context.

We wanted to fix this by creating a living, conversational narrative experience—an intelligent audiobook that dynamically adapts, feels emotion, and acts as a personalized tutor.

🚀 What it does Project Arivu is an interactive narrative learning platform. It transforms flat, static PDFs and texts into vivid, voice-interruptible audiobooks.

While listening to a document, you can physically speak to it and interrupt the narrator in real-time. If you say, "Wait, explain that part again in simpler terms", the system instantly pauses, thinks, retrieves the exact context of what you just heard, explains it to you in a conversational tone, and seamlessly resumes narration exactly where it left off.

⚙️ How we built it (Architecture & Multi-Layer Agents) We moved away from single, slow, monolithic LLM loops and built a Multi-Layered Agentic Architecture.

The "Smart Wrapper" Intelligence Layer (Gemini AI) Instead of pushing every single microphone noise to a heavy RAG pipeline, we implemented Gemini AI as our "Smart Wrapper." Think of it as the system's frontal lobe. It processes continuous live streaming audio from the user and performs high-speed intent classification.

It distinguishes between valid commands (Questions, Pause, Rewind) and random "chit-chat" or background noise. If you cough or your dog barks, the Smart Wrapper classifies it as noise and the narration continues flawlessly. When a valid question is detected it dictates exactly which downstream agent should take over.

Multi-Agent RAG & Emotional Audio When the Smart Wrapper triggers a "Question" event, our underlying Multi-Agent RAG system kicks in. We mathematically bound the semantic search so the agents can only retrieve context from chunks of the document you have already heard—preventing the AI from spoiling the end of the chapter.

Simultaneously, Emotional Sentiment Agents analyze the responses to dictate the text-to-speech parameters, dynamically shifting pitch and pacing to make the tutor sound empathetic, dramatic, or objective.

Infrastructure

Vector Search & Knowledge: Qdrant Vector Database Speed & Caching: Redis (to instantly cache chunk boundaries and generated audio) Agent Workflows: Flowise for workflow orchestration Compute: Dockerized microservices (FastAPI backend + Next.js frontend) 🚧 Challenges we ran into The absolute hardest challenge we faced was optimization for near real-time conversion and response generation.

In an auditory pipeline, latency is highly perceptible. The temporal delay $T_{\text{delay}}$ from when a user stops speaking to when the system replies is defined by: $$ T_{\text{delay}} = T_{\text{STT}} + T_{\text{Wrapper Infer}} + T_{\text{RAG}} + T_{\text{TTS Generation}} $$

If this process takes 5 seconds, the illusion of a live tutor shatters. We solved this through aggressive Parallel Just-In-Time (JIT) Pipelining and Redis caching. While the user listens to Chunk $N$, the backend is silently pre-computing and caching the semantic analysis and TTS audio for Chunk $N+1$, $N+2$, and $N+3$.

🧠 What we learned We learned that building conversational agents isn't just about LLM prompt engineering; it is deeply rooted in systems engineering. Synchronizing HTML5 audio buffers with persistent React client states, while managing asynchronous websockets for speech recognition, pushed our full-stack capabilities to the limit.