Evidence Freezer

Evidence Freezer turns AI runtime traces into security case files with Gemini-powered analysis, Phoenix observability, Firestore storage, and remediation review.

Comment

Inspiration

AI apps are becoming tool-using systems, but most security reviews still happen after the fact with scattered logs, screenshots, and guesswork. I wanted to build a workflow that treats AI observability traces as forensic evidence and turns prompt injection or RAG injection events into structured, reviewable security cases.

What it does

Evidence Freezer captures attacks against a deliberately vulnerable AI app, stores the runtime traces in Arize Phoenix, and processes them through a private watcher service. The watcher reads Phoenix trace evidence, runs detector pre-screening, sends normalized evidence to Gemini 2.5 Pro on Vertex AI Agent Engine, and stores the resulting case file in Firestore.

The dashboard lets reviewers inspect the incident type, severity, evidence pair, timeline, detector output, root cause, and proposed prompt patch. It also supports patch replay so a proposed fix can be tested against the original attack prompt.

How we built it

The stack runs on Google Cloud. The target app, dashboard, watcher, and Phoenix adapter run on Cloud Run. Phoenix stores OpenInference-style traces from the vulnerable app. The watcher uses a Phoenix MCP-compatible adapter to pull trace evidence, then calls Gemini 2.5 Pro through Vertex AI Agent Engine. Firestore stores case files and replay results.

The frontend is built with Next.js and TypeScript. The services are containerized with Docker, deployed to Cloud Run, and validated with GitHub Actions, unit tests, build checks, and GCP smoke tests.

Challenges we ran into

The hardest part was making the demo real without making it unsafe. The target app is intentionally vulnerable, so the public surface had to stay separate from private processing services. Cloud Run IAM, scheduler-triggered polling, Firestore access, Phoenix auth, and Agent Engine invocation all had to work together.

Another challenge was making Gemini output strict structured case files while still handling messy trace evidence. We added schema validation, dedupe, detector pre-screening, and replay checks to keep the workflow reliable.

Accomplishments that we're proud of

Evidence Freezer demonstrates an end-to-end AI security workflow: attack, trace capture, evidence extraction, Gemini analysis, case-file generation, Firestore persistence, and remediation review.

I am especially proud that the dashboard is not just a static mockup. It reads live Firestore case documents generated from Phoenix traces and Gemini analysis.

What we learned

AI observability data is valuable security evidence, but it needs structure. Phoenix traces can show what happened, while Gemini can help classify and summarize why it matters. The key is keeping trace content as evidence, not trusted instructions.

We also learned that private service boundaries matter. The vulnerable target app should not directly control Gemini analysis or internal processing. The watcher acts as the controlled bridge.

What's next for Evidence Freezer

Next steps are integrating the official Arize Phoenix MCP server directly, adding event-driven trace processing for near-instant case creation, improving RAG-injection coverage, and expanding patch replay into a fuller remediation workflow.

Built With

  • arize-phoenix
  • docker
  • gemini-2.5-pro
  • github-actions
  • google-cloud-firestore
  • google-cloud-run
  • google-cloud-scheduler
  • mcp-compatible-trace-adapter
  • next.js
  • node.js
  • openinference
  • phoenix-client
  • typescript
  • vertex-ai-agent-engine
Share this project:

Updates