AgentOS: The Self-Teaching Partner

An autonomous, self-documenting agent that pauses for your guidance, learns your workflows, and permanently remembers them as reusable SOPs.

Comment

💡 Inspiration

Current autonomous AI agents share a critical flaw: when they don't know how to perform a task, they either hallucinate or fail silently. We realized that true autonomy isn't about knowing everything on day one—it's about knowing when to ask for help.

Inspired by the "Collaborative Partner" track, we envisioned an agent that mimics a junior employee. When given a complex or ambiguous task, instead of guessing, it pauses, asks the human for explicit instructions (Human-In-The-Loop), executes the task, and critically—writes a Standard Operating Procedure (SOP) to its Memory Vault so it never has to ask again.

⚙️ What it does

AgentOS is a self-teaching, context-aware autonomous agent platform.

  1. Task Execution: Users submit tasks via a sleek Angular frontend.
  2. Memory Retrieval: The agent searches its Firestore Memory Vault for existing SOPs.
  3. HITL (Human-in-the-Loop): If no SOP exists, the agent halts and dynamically renders a UI prompting the user for guidance, offering clickable suggestion chips.
  4. Permanent Learning: Once guided, the agent not only completes the task but uses Gemini 3.5 to generate a permanent SOP, storing it in Google Cloud Firestore. The next time a similar task is requested, the agent completes it entirely autonomously.

🛠️ How we built it

We utilized a robust, modern tech stack deployed entirely on Google Cloud:

  • AI Reasoning: We utilized the Google Gen AI SDK with gemini-3.5-flash-lite for the core orchestration, plan generation, and SOP documentation.
  • Safety & Guardrails: Gemma 4 open models are used to sanitize inputs and validate the generated SOPs.
  • Backend: A highly concurrent FastAPI (Python) server manages state, tool execution, and the HITL API endpoints.
  • Database: Google Cloud Firestore (Native mode) acts as the permanent Memory Vault for storing the generated SOPs.
  • Frontend: A responsive Angular web application provides the chat UI, dynamically rendering the HITL intervention components.
  • Deployment: Fully serverless deployment via Google Cloud Build, Artifact Registry, and Cloud Run.

🚧 Challenges we ran into

The hardest challenge was breaking the traditional synchronous "request-response" cycle. Because the agent might run for minutes and then suddenly need human input, we had to build a robust asynchronous polling mechanism between our Angular frontend and FastAPI backend. Additionally, training Gemini to write generalized SOPs (rather than highly specific, one-time instructions) required significant prompt engineering and iterative testing.

🎓 What we learned

We learned that giving an LLM the explicit "tool" to ask for help drastically reduces hallucination rates. Furthermore, we discovered that utilizing Gemini to dynamically generate its own system prompts (in the form of SOPs) creates an incredible compounding effect where the agent becomes exponentially smarter and faster the more you use it.

🚀 What's next for AgentOS

In the future, we plan to implement "Team Vaults" where multiple agents across an enterprise can share SOPs with one another. We also plan to integrate multi-modal HITL, where the agent can send a screenshot of a failed browser automation task to the user and ask, "Where should I click next?"

Built With

Share this project:

Updates

Submission history