Model Testing Update
After running the full pipeline across multiple providers, here's what we found:
GPT-4o — Best performance. All three LLM calls (feature extraction, profile update, anomaly explanation) return well-structured JSON consistently. Flags fire reliably and explanations are coherent. Recommended for production use.
Gemini 2.5 Flash / Pro — The hosted demo runs on Gemini 2.5 Flash. It works, but JSON serialization from Call 1 is less consistent — occasionally returns malformed feature vectors that fall back to defaults. Explanations are generally good but less precise than GPT-4o.
Anthropic Claude — Solid performance, comparable to GPT-4o.
To get the best experience:
- The live demo (link above) runs on Gemini 2.5 Flash — functional but may show occasional inconsistencies
- For best results, clone the repo and run locally with GPT-4o:
bash LLM_PROVIDER=azure # or openai LLM_MODEL=gpt-4o
The system is provider-agnostic — swap models by changing two environment variables. No code changes needed.