Drug Interaction Agent

AI agent that checks drug interactions, grades its own reports with LLM-as-a-Judge, and gets smarter every run using its own Phoenix trace history.

Comment

Inspiration

Drug interactions are a leading cause of preventable hospital admissions. Most people taking multiple medications have no easy way to understand how those drugs interact with each other, and generic chatbots that just answer questions aren't enough. I wanted to build something that acts: fetches real data, synthesizes it, grades its own output, and improves over time.

What It Does

Drug Interaction Agent is a 6-tool AI agent pipeline that takes a list of medications and produces a structured 5-section safety report:

  1. Resolve - maps drug names to RxCUI codes via NIH RxNorm
  2. Check Interactions - queries each interactions with AI usage for pharmacological knowledge
  3. FDA Enrichment - fetches boxed warnings, label warnings, and FAERS adverse event reports from OpenFDA
  4. Synthesize - Gemini 2.5 Flash writes the full report using only the collected source data
  5. Evaluate - LLM-as-a-Judge scores each section on accuracy, caution, clarity, and citation (0.0–1.0), using judge criteria managed in Phoenix Prompt Management
  6. Self-Improve - sections scoring below 0.55 are rewritten with stricter prompting; final scores are written to a Phoenix dataset for cross-run learning

Each run reads historical weakness patterns from Phoenix before synthesis - so sections that consistently score low receive focused prompting on the next run. The agent literally uses its own trace history to get better.

How I Built It

Runtime: Python direct pipeline on Cloud Run (no ADK orchestration overhead - tools are called sequentially in code, with steps 2 and 3 running in parallel via asyncio.gather).

Observability: Every tool call gets its own OpenTelemetry child span. All Gemini calls are auto-instrumented via openinference-instrumentation-google-genai - prompts, responses, and token counts appear as LLM spans in Phoenix.

Phoenix MCP: Four operations run as MCP tool calls through @arizeai/phoenix-mcp (npx subprocess):

  • get-prompt-by-identifier - fetch versioned judge criteria at runtime
  • upsert-prompt - sync updated criteria when code changes, without redeploying
  • get-dataset-examples - read historical eval scores before synthesis
  • add-dataset-examples - write this run's scores after evaluation

Self-improvement loop: Sections averaging below 0.85 across past runs are injected into the synthesis prompt with their specific failure reasons from the judge's reasoning field. The agent reads its own weaknesses and targets them directly.

UI: Streamlit with a cold-start-aware initialization flow - Phoenix tracing loads in a background thread so the UI renders immediately, with the submit button enabling automatically when ready.

Challenges

Cold start latency. Importing arize-phoenix at module level loads pandas, sqlalchemy, and numpy - causing a 30-40 second blank screen on Cloud Run cold starts. Fixed by deferring the import inside setup_tracing() and running it in a background thread, with the UI polling via st.rerun() until ready.

Interaction coverage. The NIH Drug Interaction API only covers pairs where both drugs have RxCUI codes. Unresolved or OTC drugs get no coverage. Fixed by routing all uncovered pairs through a AI assisted pharmacological knowledge, deduplicating against NIH results.

What I Learned

  • Phoenix MCP is a genuinely useful pattern, having the agent query its own trace history, prompt versions, and eval datasets at runtime gives it a real feedback loop that a static RAG setup can't match.
  • openinference-instrumentation-google-genai makes the tracing story much cleaner, you get full LLM span visibility (prompt, response, token counts) without touching any of the tool code.

Tech Stack

  • Gemini 2.5 Flash via google-genai SDK
  • Arize Phoenix - tracing, Prompt Management, dataset history, MCP server
  • OpenInference - openinference-instrumentation-google-genai for auto-instrumented LLM spans
  • NIH RxNorm - drug resolution data
  • OpenFDA - label warnings and FAERS adverse event reports
  • Google Cloud Run - containerized deployment (Python 3.11 + Node.js 20 for MCP subprocess)
  • Streamlit - web UI
  • Google ADK - agent definition available as an alternative orchestration path

Built With

Share this project:

Updates