Prometheus

• 

  Prometheus: a self-improving AI research agent using Gemini + Phoenix to trace, evaluate, and improve its own reasoning.

Inspiration

Developers and tech leads spend hours making architecture decisions — reading 10 conflicting blog posts, unsure which benchmarks are real and which are sponsored content. The information exists, but synthesizing it into a trustworthy, cited, structured brief takes hours of human work.

We wanted to build an agent that does that synthesis — and then gets measurably better at it over time, autonomously. Not an agent that runs — an agent that learns from running. The name Prometheus came naturally: in Greek mythology, the titan who stole fire from the gods and gave it to humans. We're giving the agent the ability to improve itself — something previously reserved for its human operators.

Most AI agents are black boxes. When they fail — hallucinate a statistic, drop a citation, reason in circles — there's no record of why, and no mechanism to fix it without a human intervening. The feedback loop is broken by design. Prometheus fixes that.

What it does

Prometheus is a multi-step research and synthesis agent powered by Gemini 2.5 Flash and Google ADK. Ask it a hard, multi-part question. It works through six structured stages automatically:

1. Decompose — breaks the question into focused sub-problems
2. Search — runs independent web searches per sub-problem (Tavily → DuckDuckGo fallback, both free)
3. Extract — pulls structured claims from sources, classifies them by type (statistic, assertion, hedged), flags ones needing verification
4. Contradict — identifies conflicting information across sources and flags severity
5. Synthesize — produces a structured report: Executive Summary, Key Findings, Evidence & Citations, Conflicting Views, Confidence Assessment, Open Questions
6. Self-evaluate — assesses citation coverage before finalizing, flags unsupported claims

Every span of every run is traced to Arize Phoenix via OpenInference — zero manual instrumentation code. Every response is scored by an LLM-as-Judge eval pipeline. After each batch, a second improvement agent reads its own traces via the Phoenix MCP server, diagnoses failure patterns, rewrites the system prompt, and saves the new version back to Phoenix — autonomously, without human intervention.

The next run uses the improved prompt. The scores go up. The loop repeats.

How we built it

Agent runtime: Google ADK with InMemoryRunner. The research agent has five tools, all registered as FunctionTool (not ADK built-ins — mixing built-in google_search with FunctionTool causes a hard SDK incompatibility we discovered and worked around):

• search_web() — Tavily primary, DuckDuckGo fallback, no paid API required
• search_wikipedia() — free factual grounding via Wikipedia REST API
• extract_key_claims() — classifies claims by type, flags ones needing verification
• identify_contradictions() — compares claims across sources, flags numeric disagreements >25%
• assess_citation_quality() — reviews draft report before output, flags unsupported claims

Tracing: openinference-instrumentation-google-adk with auto_instrument=True via phoenix.otel.register. Every LLM call, tool call, and agent decision becomes a span in Phoenix Cloud automatically. Each research query produces 6–8 nested spans.

Evals: Custom LLM-as-Judge pipeline using google-genai directly (the arize-phoenix-evals high-level API had compatibility gaps with our version). Four evaluators: citation_groundedness, reasoning_coherence, hallucination_risk, completeness. Scores logged back to Phoenix as span annotations via client.spans.add_span_annotation() using annotation_name= (not name= — a parameter mismatch we debugged from the Phoenix source).

Self-improvement loop: A second ADK agent uses @arizeai/phoenix-mcp as its tool layer, calling list-traces, get-spans, get-prompt-version, and upsert-prompt at runtime. It reads real operational data from Phoenix, diagnoses failure patterns with specific examples from actual traces, rewrites the system prompt, and saves a versioned prompt to the Phoenix Prompts registry. Change logs are saved per cycle and are auditable.

Deployment: Streamlit web UI deployed on Google Cloud Run via gcloud run deploy --source. All keys passed as Cloud Run environment variables — no secrets in the container.

Stack: Google ADK · Gemini 2.5 Flash · Vertex

Built With

• adk
• arize
• arize-phoenix-evals
• gemini
• mcp
• openinference
• opentelemetry
• phoenix
• python
• vertex