Guardian AI

Imagine an agent who never makes you feel guilty for not showing up when your parents needed you for their Gmail password reset or to figure out which email to use for shopping on amazon.

Comment

Inspiration

Its been 3 years since I have come to Dublin, Ireland - A place I call home away from Home, coming here took me closer to a lot of things my career, love, ambition, luxury, opportunities but it sure took me away from the two pieces of my heart - Mummy and Papa(They live in India). And while away, they never gave me any pressure other than to be the best version of myself, something that my parents never hesitated to ask me was to help them with the ever-changing processes of technology. As loving and caring they are, I would be nothing to them even after completing a bachelors in engineering and a masters in AI if I cannot remote troubleshoot their amazon account logins, bank login problem, password resets, creating new passwords or remembering what emails my parents have along with their passwords (There was a time I had a note saved with 6-8 emails for them combined with their passwords - No am not kidding!).

We all must have had these moments when we were busy building our careers, studying for exams or just plain lazy to do the repeated "simple for us" tasks for them, often directing them to relatives, neighbhours and sometimes even strangers for help. So this was part one of my inspiration. My other half comes from meeting a bunch of lively men in their 50s coming from countryside Scotland to Liverpool for the "Liverpool vs who cares" match. One of the man, I kid you not, did not know how to use WhatsApp, he got it installed by his son but never used it to forward anything, he didnt know how to share pictures, how to video call. So here comes the non techies - people who may or may not have resources or line of business to learn the ever-changing, ever-increasing technology.

My inspiration comes from guilt, care and concern for the people who's "simple problems" can be solved by my Guardian Agent.

What it does

Users

It is like a tutor/assistant for the users who want to get more from the apps, at least from the popular ones. More like a friend whom they trust, who can be patient, who they know knows things.

Business

Youtube, Instagram, Chatgpt, Gemini, Tik Tok, amazon there is one thing which ties all them together in terms of business goals - Maximum user engagement. Guardian Agent solves that problem. Of course for legacy players like Gmail which has market share of approx 85% in India itself maybe it might not care for user engagement as much but exactly that is its biggest leverage to most of these other apps. Most of them require a google email login and according to my sob story that is where it all starts emails, passwords, resets and disappointments. Guardian AI ensures that it is hand-holding the user understand how to navigate through any app hence maximizing their user engagement- because how would a user spent any time on an app if they are too unaware of how to operate it confidently.

Meet Guardian Agent

The average internet user manages around 100 passwords across digital services. Studies show that 87% of users experience ‘password fatigue’, and more than half abandon accounts or create duplicate ones rather than reset credentials.

The “Guardian Agent” is a multimodal agent which will help elderly, non technical people to navigate this problem. It will act as an tutor/assistant/friend to these users to solve the issues of:

  • 1. Whenever prompted by the user to help with account login in gmail or amazon (we will only focus on these two use cases for now). The first step will be for it to check how many Gmail accounts the user has and authenticate from the user which one to use.
  • 2. If the user is confused and doesnt know which one to use, browses through emails and check on which of the email the latest account creation or transaction email was sent and determine the email to be used for further process(specially helpful in multiple email cases - When account is created by kids or any other person for the user and user is unaware of the account email). more security feature are documented and kept for future work like changing permission flow so agent first asks everytime before browsing emails
  • 3. The agent can also act as a gmail password manager but I want good security here so a MFA or similar guardrail is required future works . Now the agent can either ask the user to individually input passwords to check Gmails or help the user to reset passwords.
  • 4. After clearing the confusion on the emails the agent goes ahead and solves the e-commerce problem of login in, even read OTP from email and informs user to input it in case of reset password. It saves a lot of back and forth from user making it less frustrating and wasting time on simple things. _future work: It can help user with best promo codes application, EMI options and using EMI option at checkout. _

How I built it

Guardian Agent was built as a tool-based multimodal AI agent powered by Google’s Gemini models. The system combines conversational AI, screen analysis, and Gmail email parsing to guide users through complex login and verification workflows.

I did the brainstorming using Gemini and Chatgpt. Maybe one day I will create an agent who would hand hold me for codex and claude code but until then am happy with my two loyal friends/teammates.

AI Agent (Gemini)

At the core of the system is the Gemini model, which acts as the reasoning engine for the assistant. The Gemini agent processes user messages, decides when external tools should be used, and generates helpful responses.

We implemented this using the Google Generative AI Python SDK, allowing Gemini to manage conversation memory and trigger different tools depending on the user’s request.

Streamlit Interface

The user interacts with the assistant through a Streamlit-based chat interface. Streamlit provides a simple and interactive frontend where users can type requests, view responses, and trigger screen analysis using a button.

This interface allows the assistant to behave like a conversational support agent.

Tool-Based Architecture

Guardian Agent uses a tool-calling architecture, where the Gemini model can trigger external tools to perform specific tasks.

Three main tools were implemented:

  • 1. Gmail Tool

The Gmail tool connects to the Gmail API using OAuth authentication. It searches inboxes for Amazon emails and extracts OTP verification codes from the email body. Email content is parsed using BeautifulSoup to handle both text and HTML email formats.

  • 2. Vision Tool

The vision tool captures a screenshot of the user’s screen and sends it to the Gemini model for analysis. This allows the agent to understand what the user is currently seeing and provide contextual guidance.

  • 3. Browser Tool

The browser tool allows the assistant to open Amazon in the user’s browser when the user explicitly requests it.

Multimodal Reasoning

By combining text input with screenshot analysis, the assistant becomes a multimodal AI system. Gemini can reason about both the user’s request and the visual context of the screen.

For example, when a user writes they are stuck on an Amazon login page, the assistant can analyze the screenshot and guide the user through the verification process.

Voice Feedback

To improve accessibility for users, the assistant also provides spoken responses using the macOS speech engine, allowing users to hear instructions instead of reading them.

Challenges I ran into

There many small ones, numerous I would say but the biggest two

  1. I have a M1 2020 Macbook Air and this B*eautiful A*sset is slow and cannot handle a lot of security and aesthetics I want to include - Faster! but developing locally on an M1 Mac taught me the importance of optimizing model latency and efficient tool-calling to ensure a smooth experience on consumer-grade hardware.
  2. For a very long time I was stuck on OTP readability. (with the regex and snippet logic)
  3. I got to know about this hackathon only 4 days before the deadline so time crunch.

Accomplishments that I am proud of

It reads the OTP now!!! No am kidding this time. The ability and possibility of Guardian Agent to browse through emails and "see" the amazon page and guide user Exactly what is suppose to be done. I remember when whatsApp video calls could bring families closer. I see the same potential in Guardian Agent - bridging technology gaps between people, families, friends.

People fear AI not because one day it will diminish human value, but because they dont know how to use it for their advantage so they are not left behind.

What I learned

Sometimes simplest problems are hardest to solve. Lets just say I believe in Responsible AI and I want to make this process where user is still in power and in which the agent is it's best friend.

What's next for Guardian AI

Security Upgrades:

  • The agent can be autonomous or work only after explicit permissions from user, but the choice will be always with user.
  • Prompt injection tests.
  • Rules on chat history and password storages.

Features

  • To be extended to full app cycle - In case of amazon full shopping cycle from login - promo code suggestion - EMI options help - address & billing - informing customer when the product will arrive.
  • Extending this to have a base model under Google's umbrella of product which can be extended to any business on demand (Maximum app time by user case)
  • Speech input - I did try to add this for the demo but I was short on time.
  • design changes, interference changes.

Built With

Share this project:

Updates

Submission history