-
-
Architecture diagram
-
Architecture diagram (Databricks)
Inspiration Wacky Wheels Co. receives messy operational messages: a plant goes dark, a chip shipment slips, a supplier reports a faulty weld, plus rumors, brochure questions and a digest with two incidents at once. Someone has to work out which production orders, customer orders, dealers and owners are affected, then act in the MES, CRM, dealer and quality systems without breaking policy or following an instruction hidden in a message. We wanted a control tower where every action traces to a record and a rule, and where a language model can read the text but never acts.
What it does One Coordinator and three specialist agents (Supply Chain Planner, Customer Care and Quality, Dealer Network) work through the 15 events. They interpret each message, walk a knowledge graph built from the 16 supplied CSVs, and act through the enterprise API. Anything unconfirmed goes to a person.
The official run made 902 calls (766 writes, 134 reads, 2 escalations) with 0 failed. The 3 noise events made no calls, and the 2 unconfirmed reports were each escalated once with a specific reason. A local web UI shows the event feed, a three-part audit view for each event (including what was left unchanged), graph path diagrams, the agent timeline, the action log with a planned-versus-logged check, the escalation inbox and the tool catalog.
How we built it
Model role: the language model does one job, Stage A. It turns one event text into a JSON extraction checked against a schema. Plain code: everything else is plain Python. Stage B checks names, dates, shipments and lots, and the confirmation gate. Graph traversal records provenance on every hop. A 13-rule policy validator can deny a whole plan. Writes only happen after live reads. Verification: every event gets a trace that matches the server log line by line. Impacted sets equal an independent SQL recomputation. Databricks: Stage A runs through a Databricks provider. It sends the event text only, uses an OAuth CLI profile with no stored token, verifies TLS, and enforces a persistent call budget behind a config gate. Model choice: a bake-off compared Llama 3.3 70B Instruct and gpt-oss-20b on 48 synthetic cases plus the 15 real events, with the rule fixed before any call (126 scored calls). Llama 3.3 70B Instruct was selected. Official run: it replayed the 15 cached extractions, with 0 model calls. Challenges
Relative dates such as "next Friday" and "last Monday": we documented a convention and kept both readings in the trace. Policy ambiguities, such as what counts as an impacted order and whether transfers can happen on any day: we chose defaults and listed them as assumptions. Event text is untrusted, so injection attempts must never reach an action. The model comparison had to shrink to two candidates and one pass because of the deadline. Accomplishments
0 failed calls, planned writes equal logged writes, and every impacted set matches the SQL check. Every decision shows its evidence, its rule and what stayed unchanged. An honest evaluation: the 48 test cases were written by the team, and a sample of 10 plus the mechanical audit were reviewed by an AI assistant, not a human. The Databricks use stayed within its limits. What we learned Keep the model on the narrow reading job and let code verify and act. Write the evaluation rule before seeing numbers. Treat "do nothing" and "I am not sure" as first-class outcomes.
