Crucible

Adversarial AI agents that learn from every lie they miss.

Comment

Inspiration

My uncle runs an independent hotel chain in UAE, and recently, his listings were flooded with perfectly written one star reviews. The scary part? There were no typos and no robotic phrasing. The realization hit me hard: the Turing test is basically dead. AI-generated slop is now entirely indistinguishable from human writing. The spam filters of yesterday are useless. It became painfully obvious that only an AI can catch another AI. I had to build a system to fight fire with fire.

What it does

  1. The adversarial loop. Crucible spins up three Gemini agents and lets them fight. In each round, the Forger writes new fake reviews designed to exploit whatever the Detector currently struggles with. The Detector then judges a mixed batch of those fakes plus a held out benchmark of known real and known fake reviews, returning a structured {verdict, reasoning} for every one. After the round, the Reflector reads the OpenTelemetry traces of every failure from Arize Phoenix through MCP, distills one new detection rule, and writes that rule into a Phoenix Dataset called crucible-rules. The Detector reads its rule set back from that Dataset on every call, so by round two it is already operating with the lesson it learned in round one. The loop runs three rounds in the live demo, and you watch all of it happen in real time on screen.
  2. Auditing any hotel on the internet. Once the loop has hardened the agent, you point Crucible at any hotel by name. It calls the Google Places API to pull the live customer reviews for that hotel, runs each one through the same Detector that just finished training, and shows you the aggregate verdict (a suspicious percentage), a per review verdict, and the reasoning behind every call. If you disagree with one verdict, you click once. That click sends the failing review back through the Reflector, and a new rule lands in the Phoenix Dataset within seconds. Hit re-audit and the Detector judges the same hotel with the rule it just learned from you. Flagged reviews can be sent into a moderation queue with one approval click, so a human owns the actual takedown decision.
  3. Benchmarking against ground truth. The held out benchmark inside the adversarial loop is the OTT Deceptive Opinion Spam Corpus, an academic dataset of sixteen hundred Chicago hotel reviews, half written by real guests, half written by paid human deceivers, labelled and validated in the Cornell study that introduced the corpus. That gives Crucible a fixed scoreboard. Every round we report two numbers: Benchmark Accuracy on OTT, which proves the Detector is not drifting on the labelled set, and Adversarial Catch Rate, which proves it is keeping up with the Forger. Both numbers update live in the accuracy chart on the page. The OTT dataset is the reason the improvement is measurable instead of vibe based, and it is the same dataset most academic fake review work has been benchmarked on since 2011. The short version: Crucible learns in a sandbox against a Forger it has full control over, measures the result on a labelled academic dataset, and then applies what it learned to live, real world Google reviews with a human kept in the loop.

How we built it

Crucible is an adversarial loop with three Gemini agents and one trick. The trick is what makes them learn. The Forger generates hyper-realistic fakes designed to exploit the Detector's current blindspots. The Detector judges each review as real or fake and returns structured JSON. It has no memory of its own. On every single call it rebuilds its system prompt by pulling the latest detection rules straight from Arize Phoenix. The Reflector is a real Google ADK LlmAgent. Its McpToolset spawns the Arize Phoenix MCP server over stdio at runtime, so the agent itself decides when to call list-traces or get-trace to read its past failures. The trick is what happens after the Reflector learns. The new rule is not written to a file or our database. It is written as a row in an Arize Phoenix Dataset called crucible-rules. Every Detector call, including live audits of real Google Places reviews, reads that Dataset back from Arize before it judges. The agent cannot operate without the partner platform. Phoenix is not bolted on observability, it is the substrate the agent thinks from. Conceptually this is a minimax game in the same shape as a GAN, but the parameters being updated are prompts and accumulated rules, not model weights. There is no gradient descent. The update step is the Reflector reading traces and writing one new rule. Every Gemini call, every ADK tool call, and every MCP invocation is auto traced into the crucible project on Phoenix Cloud, which is the same place the Reflector reads from, so the system is wired into a loop with itself. It runs as a FastAPI app on Google Cloud Run, container built with a multi-stage Dockerfile that ships Python 3.11 for the agents and Node 20 for the Phoenix MCP subprocess.

Accomplishments that we're proud of

The full loop runs live in production. You can open the URL, type a hotel, and watch the Detector call Gemini, the spans land in Phoenix, and the rules update inside the partner platform in real time. That is verifiable from outside the project, not just in my recordings. The Reflector is a real Google ADK LlmAgent with an McpToolset that spawns the Arize Phoenix MCP server over stdio at runtime. It is not a wrapper. It is not a stub. It autonomously calls list-traces and get-trace, reads its own past failures, and decides what new rule to write. And the piece I am most proud of: the agent's detection rules are not in a database I control. They are not in a JSON file. They are rows in a Phoenix Dataset on Arize called crucible-rules. Every Detector call reads from Arize before it judges. Every correction writes back to Arize. The agent literally cannot operate without the partner platform. That turns observability from a passive logging layer into the substrate the agent thinks from.

What we learned

Observability isn’t just for human engineers staring at dashboards anymore. Telemetry is an agent's short-term memory. By giving an LLM the ability to read its own production traces over MCP, we moved from an agent that just acts to an agent that actually learns. What started as a quick project to help my uncle clean up his Yelp page turned into a working blueprint for self-healing AI.

What's next for Crucible

Multi-tenant memory. Right now the Phoenix Dataset is one global rule set. Adding a tenant key to each row lets every hotel chain run its own private detection brain without leaking signal across customers. Vertical transfer. The same architecture, re-grounded on product reviews, restaurant reviews, or app store reviews. The Forger, the Detector, and the Phoenix learning loop all transfer. Only the seed examples change. A smarter adversary. The current Forger is single shot per round. A Forger that targets the most recently written rule specifically, looking for the exact pattern the Detector just learned, would tighten the loop and push accuracy higher faster. A real moderation backend. The quarantine queue is in memory today. A small persistent table and a webhook into a Slack or Teams channel would make Crucible deployable inside an actual trust and safety team this week. My uncle. First production user. The whole reason this exists.

Built With

Share this project:

Updates