Azure Team

Work smart, grow fast.

Comment

Inspiration

Today's large language models are extraordinarily capable, yet they remain largely constrained within interactive chat windows. They operate in a vacuum—isolated from the developer's actual working context, local notes, active schedules, and daily habits. Most existing AI assistants are passive: they wait for a prompt, cannot verify whether their advice was implemented in practice, and lack continuous feedback loops.

We asked ourselves: What if an AI assistant didn't just chat, but understood your live context your notes, calendar, and daily focus working autonomously in the background to turn high-level goals into verified execution?

We built ContextForge as a functional Minimum Viable Product (MVP) to solve this disconnect. Our mission was to build a developer workspace that runs in the background, handles multi-source knowledge with vector search, and orchestrates real-world tasks across everyday tools (Obsidian, Google Calendar, Notion, and Android devices) using the open Model Context Protocol (MCP) with a closed-loop evaluator that continuously verifies progress toward goals.

What it does

ContextForge is a functional MVP of an autonomous agentic workspace built on 5 modular layers (Knowledge, MCP Tools, Skills & SOPs, Connections, and Agents):

1. Background Task Evaluation & Closed-Loop Goal Tracking

  • Goal Formulation: Users define high-level objectives (e.g., "Master NestJS Architecture & Complete TDD Refactor").
  • AI Task Decomposition: Powered by Gemini 3.5 Flash, ContextForge decomposes abstract goals into structured SMART sub-tasks with assigned risk levels and target MCP tools.
  • Nightly Background Evaluation via CRON: A background daemon runs autonomously every night. It gathers telemetry, verifies task completion in connected tools (Notion / Google Calendar), and calculates daily compliance metrics: $$\text{Compliance Score} = \left( \frac{N_{\text{completed}}}{N_{\text{total}}} \right) \times 100\%$$ flagging schedule bottlenecks and adapting future recommendations without manual user intervention.

2. Multi-Source Knowledge Ingestion & pgvector RAG

  • Knowledge Vault Ingestion: Ingests Markdown vaults, technical documentation, code snippets, and research materials.
  • Vector Semantic Search: Generates 1536-dimensional embeddings with gemini-embedding-2 stored in PostgreSQL with pgvector, queried directly via native connection pools with zero ORM overhead: $$\text{Cosine Similarity} = \frac{\mathbf{u} \cdot \mathbf{v}}{|\mathbf{u}| |\mathbf{v}|}$$

3. Multi-Tool Orchestration via Model Context Protocol (MCP) Gateway

ContextForge natively implements the open Model Context Protocol (MCP) across 4 functional connectors:

  • ** Obsidian Vault Bridge**: Local note reading, bi-directional [[wiki-link]] traversal, and YAML frontmatter preservation.
  • ** Google Calendar Mutator**: Event scheduling, checking free/busy slots, conflict detection, and calendar rebalancing.
  • ** Notion Workspace Integration**: Two-way database syncing and automated reflection note creation via OAuth 2.0.
  • ** Android Bridge**: Zero-scraping ambient focus telemetry (UsageStatsManager, ZenMode) for deep-work focus enforcement.

4. Human-In-The-Loop (HITL) Safety Architecture

  • Dual-Agent Safety Pattern: The primary conversational orchestrator remains strictly read-only for safe planning and reasoning. Any mutating operation generates an interactive Action Card that requires explicit user confirmation before execution side-workers mutate physical data.

How we built it

ContextForge is engineered from the ground up as a modern, type-safe fullstack system:

  • AI & Reasoning Core: Powered by the Google GenAI SDK (@google/genai) with Gemini 3.5 Flash for multi-step tool execution and plan formulation, supported by gemini-3.6-flash and gemini-embedding-2.
  • Backend API & Event Engine: NestJS 11 and TypeScript 5.7+ featuring a decoupled domain event bus (EventEmitter2), Server-Sent Events (SSE) for real-time thought timelines, and background CRON schedulers.
  • Database & Vector Engine: PostgreSQL 16+ with the pgvector extension and native pg.Pool connection pooling — eliminating ORM latency and ensuring strict parameterization safety.
  • Frontend Client: React 19, Vite 8, Tailwind CSS 4, and Motion for an editorial, responsive workspace interface.
  • Cloud-Native Deployment: Packaged as a single multi-stage Dockerfile (~130 MB) running on Google Cloud Run with scale-to-zero configuration to keep idle infrastructure costs minimal.

Challenges we ran into

  1. Preventing Agent Hallucinations on Task Completion (Zero-Assumption Verification): A common failure mode in LLM agents is "premature satisfaction" — the model assumes an action succeeded simply because it called a tool. To solve this, we engineered a strict Zero-Assumption Verification Policy. The agent is prohibited from self-reporting task completion; instead, it must independently query live external APIs (e.g., inspecting actual Notion page status or validating Google Calendar response IDs) before updating goal progress.

  2. Normalizing Data Across Heterogeneous MCP Tool Connectors: Orchestrating four radically different tool ecosystems (local markdown filesystem for Obsidian, OAuth 2.0 REST for Calendar & Notion, and WebSocket telemetry for Android) through the open Model Context Protocol was technically demanding. We had to build robust schema parsers and defensive fallback handlers to gracefully handle mobile disconnections, calendar timezone discrepancies, and bi-directional markdown link resolutions.

  3. Balancing Real-Time Thought Streaming with Human-In-The-Loop (HITL) Safety: We wanted the user to see the agent's thought process live via Server-Sent Events (SSE) without giving the agent unrestricted write access to personal data. Designing the dual-agent architecture — where a read-only Orchestrator reasons in real-time, but destructive mutations pause for interactive Action Card confirmation — required careful state synchronization across the NestJS event bus and React frontend.

  4. Fullstack Single-Container Cloud Run Optimization: Packaging both the NestJS API engine and the React 19 SPA into a single lightweight Docker container (~130 MB) running under 512 MB RAM on Google Cloud Run required meticulous multi-stage caching and clean cold-start connection pool management.

Accomplishments that we're proud of

  • Functional Closed-Loop Agentic Loop: Built an end-to-end system where the agent autonomously evaluates scheduled goals in the background and reports progress grounded in real-world tool telemetry.
  • Working MCP Integration Across 4 Connectors: Connected Obsidian, Google Calendar, Notion, and Android within a unified Model Context Protocol layer — enabling the agent to read and act on real data from tools developers use daily.
  • Serverless Fullstack Deployment: Successfully deployed the entire fullstack application (NestJS API + Vite SPA) to Google Cloud Run with scale-to-zero to minimize running costs.
  • Type-Safe, Modular Architecture: Built a clean codebase with 8 automated test suites verifying tool schemas, background schedulers, and goal evaluations without external mock dependencies.

What we learned

  • Autonomous agents become significantly more useful when they transition from conversational assistants to telemetry-grounded executors with real-world tool awareness — the ability to verify outcomes is what separates a useful agent from a clever chatbot.
  • Separating planning (read-only orchestrators) from physical execution (ephemeral side-workers with HITL safety gates) is essential for building safe and reliable autonomous systems.
  • Adhering to the open Model Context Protocol (MCP) standard radically simplifies multi-platform integration compared to building custom, brittle point-to-point APIs for each tool.

What's next for Azure Team

As we expand ContextForge beyond this MVP, our development roadmap includes:

  1. System Hardening & Data-Driven Goal Adaptation:

    • Automated Goal Self-Correction: Enabling the AI to re-calibrate unrealistic goals, adjust milestone deadlines, and restructure failed tasks based on historical compliance trends and burnout signals.
    • Proactive Interventions: Moving beyond passive logging to trigger proactive suggestions (e.g., auto-suggesting schedule rebalancing when consecutive task friction is detected).
    • Self-Healing Error Recovery: Upgrading side-workers with automated retry strategies when third-party MCP APIs fail or hit rate limits.

  2. Ecosystem Expansion with Additional MCP Connectors:

    • GitHub & GitLab MCP: Autonomously verify code pull requests, commits, and issue closures against active engineering goals.
    • Linear, Jira & ClickUp MCP: Two-way sprint ticket ingestion and progress synchronization for developer workflows.
    • Slack & Discord Dispatchers: Delivering async morning briefings and nightly reflection summaries directly into developer channels.
    • Custom Remote MCP Registry: Supporting community-contributed MCP servers via standard STDIO/SSE streams.

  3. Gemini Live API Voice Debriefing:

    • Implementing bidirectional, low-latency voice streaming for conversational hands-free daily planning and evening reflection sessions.

  4. Multi-Tenant Collaborative Team Workspaces:

    • Enabling distributed teams to share central vector knowledge vaults and assign collaborative goals across peer agents.

Built With

Share this project:

Updates

Submission history