Smart Email Manager (SEM) Agent

Master your inbox with Smart Email Manager! Gemini 3 powered agent that autonomously organizes mails by semantic integrity loops. No more clutter, just pure AI guarding your productivity.

Comment

Smart Email Manager (SEM): A Gemini 3 Powered Agentic Suite

Inspiration

Emails are the lifeblood of modern communication, but as our inboxes grow, the burden of manual organization becomes overwhelming. Traditional folders and filters often lead to cluttered, lackluster systems where critical opportunities, from professional windows to quality-of-life offerings, are easily missed. SEM was born from the frustration of decades of manual sorting, aiming to replace tired reorganization habits with an intelligent, autonomous manager that truly "understands" your data.

What It Does

Smart Email Manager is an automated, personalized inbox organizer that relieves users of the manual labor of sorting mail. Built as a Gmail Add-on powered by Gemini 3 and Vertex AI Agent Builder, SEM categorizes your emails into meaningful themes while keeping your data strictly private. By following a "Bring Your Own Infrastructure" (BYOI) model, it runs on your own GCP environment (Cloud Run, Pub/Sub, MongoDB Atlas), ensuring total control and low-cost maintenance.

How We Built It

The system follows a sophisticated architecture divided into three primary semantic paths:

  1. Mail Ingestion: A Gmail Watcher (Apps Script) signals new mail via Pub/Sub. The Cloud Run agent fetches the content, generates high-quality embeddings via Voyage AI, and stores the record in MongoDB. By default, new mail is tagged as "sem_unclassified."
  2. Semantic Clustering & Label Creation: When the system initializes or themes shift, SEM performs K-Means clustering on unclassified mail. It selects the top 5 snippets closest to each cluster centroid, which Gemini 3 uses to brainstorm concise label names. At birth, each label is assigned a "Semantic Integrity Score" and an adaptive threshold (85% of its birth score).
  3. Real-Time Vector Classification: To ensure efficiency, incoming "unclassified" mail is periodically checked against existing label centroids using MongoDB Vector Search. If an email strongly matches an existing theme, it is filed instantly.
  4. Degradation Management: The agent constantly monitors label health. If new mail pulls a label’s integrity below its adaptive threshold, a "Self-Healing" loop triggers a targeted reorganization of that category.

Challenges We Ran Into

  1. Security & Integration: Navigating the complex security requirements of the Gmail API and GCP sensitive scopes required innovative workarounds to maintain a smooth user setup.
  2. Cloud Orchestration: Finding the right balance among the vast array of GCP services and managing manual activations (like Vertex AI Platform) for a cohesive agentic flow.
  3. Gemini CLI Collaboration: Leveraging the Gemini CLI required a disciplined, controlled review process to translate high-level architecture into stable code.
  4. BYOI Constraints: Transitioning from complex OAuth flows to simplified API key management for MongoDB and Voyage AI while satisfying the hackathon's low-cost infrastructure requirement.
  5. Publishing Barriers: Google Workspace review constraints necessitated a pivot to a robust Cloud Shell-based deployment strategy.

What We Learned

  1. SaaS Plugin Design: Architecting high-permission plugins within the Google Workspace ecosystem.
  2. Applied Machine Learning: Practical implementation of K-Means clustering and LLM-driven label generation for real-world data.
  3. Agentic AI Structure: Mastering the Model Context Protocol (MCP) to bridge LLMs with functional tools and external APIs.
  4. NoSQL Power: Utilizing advanced MongoDB features like Vector Search and Aggregation Pipelines for semantic storage.
  5. Architectural Realism: Navigating the gap between ideal software design and the technical constraints of live cloud implementation.

What We Accomplished

We delivered a fully autonomous, Gemini 3-powered Agentic AI that masters inbox management through semantic integrity scoring. It is not just a filter; it is a productivity-boosting intelligence that guards the health of your digital life.

What's Next for Smart Email Manager

Our goal is to transition from a Cloud Shell setup to a single-click deployment on the Google Workspace Marketplace and Cloud Marketplace. We aim to scale the user base while continuing to improve the agent's ability to handle complex edge cases. SEM will remain an open-source project, and we are actively seeking sponsors and contributors to help evolve this vision of an AI-managed inbox for everyone.

Built With

Share this project:

Updates