Homepage
Onboarding
Dashboard
Browser Extension
Diagram Example
Tone+Speakers Example

Inspiration

More than 5 to 7 percent of people globally, including nearly 1 in 10 children, experience Auditory Processing Disorder or similar auditory processing challenges. They can hear sounds clearly, but their brain struggles to organize, interpret, and respond to them, especially in noisy environments, fast conversations, or digital learning spaces.

What inspired Cognivo was realizing that most accessibility tools focus on accommodation rather than growth. They repeat or simplify information, but rarely help users improve how their brain processes sound over time.

We started by asking a different question: what if AI could meet someone exactly where their processing breaks down, in real time, instead of giving everyone the same generic simplification?

Cognivo is built on the belief that accessibility tools should be judged by how well they adapt to a real, specific struggle, not by how many features they pack in.

What it does

Cognivo is an AI-powered listening and comprehension assistant for people who find it hard to keep up with spoken content in real time, available as both a web app and a Chrome extension.

It starts with a short onboarding flow that identifies the user's specific struggle, whether that's missing key terms, losing the thread mid-sentence, forgetting steps, or hearing words without absorbing meaning, and uses that profile to shape everything downstream.

From there, Cognivo:

Transcribes live audio in real time, streaming microphone or tab audio to Gemini's Live API over a WebSocket connection, with automatic reconnect and backoff if the connection drops.
Explains, not just summarizes, generating simplified explanations tuned to the user's onboarding profile rather than a one-size-fits-all summary.
Answers follow-up questions in context, so a user can ask "wait, what did that mean?" about anything already explained.
Decides when a visual would help, and when it does, generates a constrained, genuinely legible Mermaid.js flowchart or comparison diagram instead of a wall of text.
Checks comprehension, auto-generating a short quiz based on what was just explained.
Reads explanations aloud with tone-aware, multi-speaker narration, and highlights each word in sync with the audio as it plays, so users can follow along visually and audibly at once.
Manages AI usage fairly, routing every request through a tiered model system (four quality tiers per feature) that automatically falls back to a lighter model instead of hard-failing once a tier's daily quota is used.

How we built it

Cognivo is a full-stack, AI-first system with accessibility and real-time responsiveness at its core, shipped as both a web app and a Chrome extension sharing one backend.

Frontend

Built with React, Vite, and Tailwind CSS, with a strong focus on reducing cognitive load. Every element, from font choice (including OpenDyslexic and Atkinson Hyperlegible) to color mode to layout, was built to stay out of the user's way rather than compete for their attention.

Backend

Built on Fastify with MongoDB for persistence. The backend owns onboarding profiles, quota state, and the real-time audio pipeline. A WebSocket layer bridges the browser and Gemini's Live API, with reconnect logic that backs off exponentially on repeated failures but resets once a session stays healthy, so a flaky connection degrades gracefully instead of hammering the API.

A tiered quota engine sits in front of every AI feature (explanations, diagrams, live audio, quizzes), tracking usage per user, per feature, and per model tier, and automatically downgrading to a lighter model rather than failing outright once a tier is exhausted.

AI and Intelligence Layer

Cognivo is powered by Google Gemini, split intentionally across tasks by strength. Gemini's newer reasoning-tier models handle explanation generation, tone-aware TTS scripting, and diagram planning, while a stable, lightweight model handles native audio transcription. A separate live-audio model powers the WebSocket transcription stream.

Because Gemini's TTS API doesn't return word-level timestamps, we built our own timing estimator that spreads playback duration across words, weighted by word length and punctuation, to drive the read-along highlighting.

Challenges we ran into

Keeping a live audio WebSocket connection reliable across real network conditions, not just the happy path, without either silently failing or reconnect-spamming the API.
Designing a diagram generator that actually helps rather than adds noise. Early Mermaid output was full of cut-off phrases and cluttered nodes; we had to constrain node count, word count, and phrase completeness directly in the prompt to get something genuinely readable.
Balancing four model tiers per feature so heavier users get a smooth downgrade experience instead of an abrupt "quota exceeded" wall.
Solving word-level audio highlighting without any native timestamp support from the TTS API.
Coordinating a shared backend across two very different clients, a full web app and a Chrome extension, under real time constraints.

Accomplishments that we're proud of

A real, working real-time audio pipeline: WebSocket streaming, transcription, and reconnect handling that survives real-world network flakiness.
A tiered, quota-aware model routing system that fails gracefully instead of failing hard.
A diagram generation pipeline that produces genuinely legible, accessibility-focused visuals instead of decorative clutter.
Word-synced narration built entirely from estimation, with no native timing support from the underlying TTS API.
Shipping both a full web app and a Chrome extension on one shared backend within a tight timeframe.

What we learned

How to design and ship a real-time, full-stack AI application end to end, not just a request/response wrapper around an LLM.
How to build browser-based experiences, including a Chrome extension, on top of the same backend as a web app.
How to route work across multiple Gemini models deliberately, matching each model's strengths (reasoning, audio understanding, live transcription, TTS) to the right part of the pipeline.
How much prompt-level constraint (word limits, phrase-completeness rules, node caps) it actually takes to get an LLM to produce something as simple-sounding as "a readable diagram."
How accessibility and AI genuinely intersect when you build for one specific struggle instead of a generic audience.

What's next for Cognivo

The next step is real-world feedback from people who actually experience auditory processing challenges, to validate what's helping and what isn't.

Longer term, we want Cognivo to move from assisted listening toward genuine independence: tracking behavioral signals like listening speed, pause frequency, and how often a user needs a simplification, so the system can gradually step back once a user starts succeeding on their own, rather than assisting at a fixed level forever. Alongside that, we're prioritizing proper authentication and account security, automated testing, and making the quota and session state resilient across multiple backend instances, so Cognivo is ready for real users, not just a hackathon demo.

Cognivo is about giving people the freedom to learn, communicate, and live without being limited by how their brain processes sound.