Inspiration
KindlyClick came from a very personal and recurring experience: being “the computer guy” in the family.
I have often helped seniors around me who are perfectly capable, thoughtful people, but who struggle with modern computer interfaces because those interfaces assume a lot of invisible knowledge. Something as simple as a modal on a website can be deeply confusing: many seniors interpret it as a system popup rather than part of the webpage itself. The same goes for tabs, windows, overlays, side panels, and dynamic UI elements that appear and disappear without clear explanation.
What I kept seeing was not a lack of intelligence or willingness, but a mismatch between how modern interfaces behave and how many non-technical users understand them.
And there is another problem: human help does not scale. I can sit next to someone and explain where to click, what a tab is, or why a popup is not “the computer blocking them” — but I am not always available. I wanted to explore whether an AI assistant could become a patient, real-time guide inside the browser: something that can listen, look at the page, explain what is happening, and point the user in the right direction.
KindlyClick is the beginning of that idea.
What it does
KindlyClick is a multimodal browser assistant designed to help seniors and low-confidence computer users navigate websites more comfortably.
It can:
- listen to the user’s spoken question
- observe the current browser page
- understand what is visible on screen
- answer with voice
- visually highlight the relevant part of the page
Instead of giving abstract instructions like “go to the top right and find settings,” KindlyClick aims to make guidance concrete and immediate. The goal is not just task completion, but confidence-building.
A user should be able to ask things like:
- “Why can’t I click anything?”
- “What is this popup?”
- “Where do I need to click to send this?”
- “Which tab am I supposed to use?”
How I built it
I built KindlyClick as a live multimodal agent centered around Gemini and Google Cloud.
At a high level, the system combines:
- a browser-based frontend for interaction
- real-time voice input and audio output
- page understanding through live screen/context sharing
- a backend agent layer using Google’s GenAI tooling
- Google Cloud services for deployment and supporting infrastructure
The app is built around the idea that helping with UI requires more than text chat. A useful assistant for this problem must be able to:
- hear the user,
- see the interface,
- reason about what is happening,
- respond naturally,
- guide attention visually.
That is why the multimodal aspect was essential from the start. This project is not meant to be “a chatbot in a browser tab,” but a first step toward an assistant that can truly teach by observing and guiding.
Challenges I ran into
The biggest challenge was making the assistant feel genuinely useful for navigation rather than merely impressive.
It is easy to build something that talks. It is much harder to build something that can help in the exact moment a confused user is blocked by the UI.
A few concrete challenges stood out:
1. Understanding real UI confusion
The problem is subtle. Seniors are often not asking for “technical support” in formal terms. They are confused by concepts that experienced users barely notice anymore:
- what belongs to the browser vs the website
- whether something is a popup, a modal, or a new page
- whether a task continues in the same tab or another one
- whether the system is “stuck” or simply waiting for input
Designing for this required focusing on very practical, moment-to-moment friction.
2. Making multimodal interaction feel natural
For this use case, the assistant has to be interruptible, conversational, and grounded in what is visible right now. If the user cannot naturally ask follow-up questions or correct the assistant in real time, the experience becomes frustrating very quickly.
3. Turning understanding into guidance
It is not enough for the model to understand the page internally. The user needs help in a form they can act on immediately. Highlighting the relevant area and combining that with spoken explanation was an important part of making the interaction usable.
4. Building something meaningful under hackathon time pressure
Like many hackathon projects, there is always tension between adding features and polishing the core experience. I had to stay disciplined and keep the focus on the main story: helping someone who is blocked by the browser right now.
What I learned
This project reinforced a few things for me.
First, accessibility is not only about screen readers, contrast ratios, or standards compliance. It is also about cognitive clarity. Many people struggle not because the interface is technically inaccessible, but because it is conceptually hostile.
Second, multimodal AI becomes much more compelling when it is tied to a real human problem. In this case, voice + vision + contextual guidance are not gimmicks; they are exactly what the problem demands.
Third, there is a lot of room for AI to act not just as an answer engine, but as a patient teacher. That is the direction that excites me most.
What’s next
I see KindlyClick as an early prototype of a much broader idea.
Today, it focuses on helping users in the browser. Tomorrow, I would love to evolve it into a desktop assistant that can observe interactions across applications and patiently teach seniors how to do everyday digital tasks step by step.
That could include things like:
- managing tabs and windows
- understanding login flows
- navigating settings
- composing and sending emails
- dealing with popups and confirmations
- learning repeated digital routines over time
My hope is that software can become more patient, less intimidating, and more humane.
KindlyClick is a first step in that direction.
Why this matters
Technology is supposed to empower people, but for many seniors it often does the opposite: it creates dependence on whoever happens to be nearby and knows how computers work.
I built KindlyClick because I want to reduce that dependence.
Even if this version is only an early start, the vision is clear: give people a calm, respectful assistant that helps them understand what is on screen and how to act — not by taking control away from them, but by teaching them patiently, one interaction at a time.
Built With
- cloud-build
- cloud-run
- firestore
- gemini-live
- node.js
- terraform
- vertex
Log in or sign up for Devpost to join the conversation.