Gemini Traceable Tool Agent

A Google Gemini agent whose every tool call is traced and evaluated in Arize Phoenix, and inspectable via the Phoenix MCP server.

Comment

Inspiration

What it does

How we built it

Challenges we ran into

Accomplishments that we're proud of

What we learned

What's next for Gemini Traceable Tool Agent

Inspiration

Most AI agents are black boxes: you can't see why they chose a tool, what it returned, or whether the answer is reliable. We wanted a Gemini agent that is traceable, evaluable, and inspectable by design.

What it does

Give it a research-to-action task. Google Gemini decides which tools to call (a calculator, a GitHub repo inspector, and a tool that inspects the agent's own Phoenix traces), a guarded dispatcher validates arguments / enforces timeouts / fails closed on unsafe input, and the model writes a final report. Every LLM and tool call becomes an OpenInference span in Arize Phoenix.

How we built it

Python google-genai with a manual function-calling dispatcher (we own tool execution for control and honest evals); arize-phoenix-otel + OpenInference for tracing (automatic Gemini spans plus manual tool spans); a deterministic golden-dataset eval (tool_call_accuracy 1.0, safety_refusal 1.0); Gradio on Hugging Face Spaces; and the Phoenix MCP server for IDE-side trace inspection.

Partner integration (Arize / Phoenix)

The agent emits real traces to Phoenix Cloud and can inspect them in-app; the same project is queryable from Claude Desktop / Cursor via the Phoenix MCP server (config included in the repo's mcp/ folder).

Challenges we ran into

Proving observability and MCP without leaking secrets; protecting a capped public key with a login-gated demo and daily limits; keeping tool-accuracy evals honest (the final report is the model's last turn, not a fake tool).

Accomplishments

A live, gated demo; real Phoenix traces with LLM and tool spans; a reproducible eval; and a clean open-source repo any team can self-host with their own key.

What's next

More tools, a larger eval set, a Cloud Run deployment, and richer Phoenix MCP feedback loops.

Built With

  • arize-phoenix
  • gemini-function-calling
  • google-gemini
  • gradio
  • hugging-face-spaces
  • openinference
  • opentelemetry
  • phoenix-mcp
  • python
Share this project:

Updates