GEODE

• 

Inspiration

AI agents have become remarkably capable at writing code, but most real-world work is broader than coding. It involves researching information, operating tools, managing files and schedules, communicating across services, remembering context, and safely completing tasks that may take hours rather than minutes.

While building agent systems, we repeatedly encountered a second problem: improving the underlying model was not always possible, but improving the system around the model was. Prompts, tool descriptions, decomposition policies, memory, reflection, and verification all strongly influenced whether an agent completed a task reliably.

GEODE began with a simple question:

Can an autonomous agent improve the scaffolding it runs on, without changing the model weights and without sacrificing safety?

That question led us to build GEODE as two connected loops: an inner loop that performs user tasks and an outer loop that experiments on the agent system itself.

What it does

GEODE is a general-purpose autonomous execution agent with a non-parametric self-improving loop.

Users describe a goal in natural language. GEODE plans the work, selects tools, observes their outputs, updates its approach, and continues until the task is complete. It supports both one-shot commands and long-running sessions backed by a persistent daemon.

GEODE can:

• Search and synthesize information from the web
• Read, create, and modify files
• Use native tools and external MCP servers
• Work with Gmail, Calendar, Drive, Docs, Sheets, Tasks, and Contacts through user-owned Google OAuth
• Run scheduled or event-driven tasks
• Operate through a terminal, Slack, Discord, or Telegram
• Delegate work to isolated sub-agents
• Preserve user, organization, project, and session context through tiered memory
• Track token usage, execution cost, tool results, and verification outcomes
• Expose itself as an MCP server for clients such as Claude Code, Claude Desktop, and Cursor

Its fundamental execution primitive is a while(tool_use) loop:

1. The model evaluates the current state.
2. It selects and invokes a tool.
3. GEODE records and validates the result.
4. The result is returned to the model as a new observation.
5. The loop continues until the model produces a final answer.

The distinguishing component is GEODE’s outer Self-Improving Loop. Instead of fine-tuning or modifying model weights, it proposes changes to the surrounding scaffold, including:

• System prompts
• Tool-use policies
• Tool descriptions
• Task decomposition strategies
• Reflection instructions
• Runtime skills
• Agent contracts

Each candidate is tested through adversarial, multi-dimensional safety evaluation. A candidate is promoted only when it demonstrates a real improvement without falling below critical safety thresholds. Otherwise, it is rejected and the previous champion remains active.

How we built it

GEODE is primarily implemented in Python, with TypeScript used for several supporting interfaces and integrations.

The system is organized as a four-layer stack:

Model layer Adapters connect GEODE to Anthropic, OpenAI, OpenAI Codex subscription routing, and ZhipuAI GLM models.

Runtime layer The runtime provides native tools, the MCP catalog, skills, memory, plans, scheduling, and service integrations.

Harness layer The harness manages session lanes, task graphs, policies, hooks, verification, isolation, and execution governance.

Agent layer The agent layer contains the main agentic loop, sub-agent manager, CLI interface, messaging pollers, and daemon gateway.

A five-tier context system assembles the relevant state for every model call:

• Tier 0: agent identity and invariant constraints
• Tier 0.5: user profile
• Tier 1: organization-level context
• Tier 2: project memory
• Tier 3: active session context

For self-improvement, we implemented a (1+1) champion-chain process:

1. Start with the current champion scaffold.
2. Generate one candidate mutation.
3. Run the candidate against adversarial evaluation seeds.
4. Apply hard safety floors and promotion gates.
5. Promote the candidate only when it produces a measurable gain.
6. Revert automatically when it fails.

The adversarial seeds also evolve through a co-scientist-style generation and ranking pipeline. This prevents the system from optimizing indefinitely against a static test set. At the same time, a separate version-frozen held-out benchmark is preserved across generations. Only improvements on this frozen set are treated as evidence of genuine progress.

We also built public benchmark adapters and preserved full execution artifacts so results can be reproduced and inspected rather than reported as isolated headline scores.

Challenges we ran into

The hardest problem was not making the model call tools. The difficult part was making tool use reliable across long, compound tasks.

An agent may correctly perform most of a workflow while omitting one required write operation, database side effect, confirmation, or user action. In those cases, the response can appear convincing even though the actual task state is incomplete. Our Tau2 experiments showed that failure frequently came from missing required actions within compound tasks rather than from a complete inability to use the available tools.

Self-improvement introduced another challenge: distinguishing real progress from evaluation overfitting.

When both the agent and its test cases evolve, a higher score does not automatically mean the system has become better. It may simply have adapted to the current evaluator. We addressed this by separating three roles:

• Mutable training and selection seeds
• Adversarial safety audits used for promotion
• A version-frozen held-out benchmark used for cross-generation comparison

We also had to design promotion rules that could not trade critical safety properties for a higher average score. GEODE therefore uses hard floors on important dimensions rather than relying only on a single aggregate reward.

Other challenges included:

• Keeping sub-agents isolated while allowing them to inherit useful capabilities
• Preventing unexpected cross-provider costs during failover
• Supporting both API-key and subscription-based model routes
• Preserving context without allowing memory to grow without bounds
• Handling partial tool failures and malformed outputs
• Making long-running daemon execution observable and recoverable
• Separating secrets, user data, project configuration, and runtime state
• Producing benchmark results with enough metadata to remain reproducible

Accomplishments that we’re proud of

We are proud that GEODE evolved from an experimental agent loop into an installable, end-to-end autonomous execution system.

Major accomplishments include:

• Building a unified while(tool_use) runtime that powers normal tasks, plans, batches, and sub-agents
• Implementing a complete non-parametric self-improving loop with mutation, adversarial audit, promotion, and automatic rollback
• Introducing co-evolving adversarial seeds while maintaining a separate frozen held-out benchmark
• Supporting Anthropic, OpenAI, OpenAI Codex subscription routing, and ZhipuAI GLM behind a common runtime
• Implementing persistent multi-tier memory and daemon-backed long-running execution
• Providing native and MCP-based tools for research, files, scheduling, messaging, and workspace operations
• Exposing GEODE itself as an MCP server
• Publishing GEODE as the geode-agent Python package with a guided setup and diagnostic workflow
• Preserving complete benchmark transcripts and verifier outputs in a public evaluation-artifact repository

In a reproducible Tau2 native user-simulator run, GEODE completed 228 of 278 tasks for a weighted reward of 0.8201 across airline, retail, and telecom domains.

In the locally available MCPMark Verified service slices, GEODE passed 64 of 74 tasks for 86.5% measured accuracy, including:

• Filesystem: 25 of 30
• PostgreSQL: 20 of 21
• GitHub: 19 of 23

We are especially proud that these results are accompanied by the exact runtime versions, model routes, benchmark commits, configuration details, per-task results, and full transcripts needed to interpret them honestly.

What we learned

The central lesson was that an agent is not just a model.

Model capability matters, but autonomous performance emerges from the interaction between the model, tools, descriptions, policies, memory, verification, context management, and execution environment. Changing the scaffold can substantially alter the behavior of an unchanged model.

We also learned that tool availability and task completion are different metrics. Giving an agent access to every required tool does not guarantee that it will perform every required action. Reliable agents need state-aware verification that checks what actually changed in the environment.

Another lesson was that average reward is insufficient for safety-sensitive improvement. A candidate that improves nine dimensions while catastrophically regressing one critical dimension should not be promoted. Explicit safety floors and rollback mechanisms must be part of the optimization process itself.

Finally, self-improvement requires an honest evaluator. Mutable tests are useful for creating new pressure, but they cannot independently prove progress. Frozen held-out evaluation, reproducible artifacts, and visible failure analysis are necessary to prevent the system from merely learning its own grading process.

What’s next for GEODE

Our next goal is to turn the experimental self-improving loop into a more rigorous and continuously measurable system.

We plan to:

• Expand the frozen held-out benchmark across more safety and execution dimensions
• Improve required-action coverage for compound tasks
• Add stronger state-based verification for external side effects
• Calibrate multi-provider judges and measure evaluator agreement
• Develop more diverse mutation operators for prompts, skills, policies, and tool contracts
• Improve causal attribution between a scaffold mutation and the behavior it changes
• Add stronger protection against evaluator overfitting and reward hacking
• Expand reproducible benchmark coverage beyond the currently available Tau2 and MCPMark tracks
• Improve long-running recovery, checkpointing, and resumability
• Provide clearer visualizations of generations, mutations, evaluation transcripts, and promotion decisions
• Make the self-improving workflow easier for other developers to reproduce on their own tasks and models

The long-term vision is not an agent that rewrites itself without limits. It is an agent that can propose bounded improvements, test them under explicit constraints, preserve evidence, and adopt them only when they are demonstrably safer or more reliable.

GEODE is our attempt to make self-improvement an observable engineering process rather than an unverifiable claim.

Built With

• python