Case Board
Live Monitor
Blue Agent Gate
Findings

Inspiration

A detective does not hand you a dashboard. A detective hands you a case file with the suspect, evidence, here is what we ruled out, here is what we still cannot prove.

The part that actually drove us was a worry. Catching a breach you were told about proves nothing. Finding the incident planted in the CSE dataset was table stakes, so we decided the real project was proving our detector would catch the next one, the one nobody described to us.

What it does

Minny takes raw web access logs and produces three things that share one evidence base.

The case file names the attacker, the victim, the asset and the method. Every claim expands to the exact raw log lines that prove it, not a summary of the lines but the bytes. It also ships the leads we checked and cleared, each with the saved query that killed it, and the three questions the logs genuinely cannot answer.

The watchdog learns what normal looks like for each person: which machine they use, which files they can open, what they usually do. Eight signals raise one plain English alarm when that pattern breaks, and related alarms collapse into a single incident, so an analyst reads one story instead of twenty warnings.

The stress test is what makes the other two believable. A generator invents two hundred variations of the attack, slower password guessing, a different victim, renamed parameters, a lunchtime run, and injects them one at a time into the same month. A blue agent proposes new detection rules for whatever slips through, and a validation gate refuses any rule that cannot catch attacks it has never seen.

On the CSE logs, 180,800 lines covering ten employees from August 2025 to March 2026, it found this. An employee sent three malformed posts to the company forum. The first two returned the only 400 and the only 500 in the entire file. The third worked. Eighteen minutes later an administrator opened forum post 1042, and one second after that her session made the only privileged admin call in 180,800 lines. Nineteen minutes later the first employee downloaded a confidential financial draft he had been refused 77 times. That night her account logged in from his workstation and took it again.

How we built it

Python 3.10, FastAPI with server sent events, pandas and pyarrow over a parquet event store, and a front end in plain HTML, CSS and JavaScript with no build step. Claude Opus 5 does the reasoning, Composio carries Slack, Gmail and GitHub, Elastic holds the indexed events and the translated queries, and Sentry traces the slow paths.

The parser is a single regex over Common Log Format that asserts zero unparsed lines and an exact row count before anything downstream runs. It normalises numeric object IDs into three template families, so per user baselines key on a template rather than on every distinct forum post. Each row carries a stable one indexed line number, and that number is the universal evidence identifier: every alert, finding, incident and generated variant cites evidence by line, which is why any claim anywhere in the product expands into the original file.

Baselines are fitted strictly below a March cutoff, compared as timezone aware instants rather than strings. That cutoff is load bearing. Built naively, the attacker counts as an authorized reader of the file he stole, because his one successful download is a 200 like any other.

Eight signals run over a replayed stream with bounded rolling state: source address against the account's baseline addresses, a first success on a path the account has only ever been denied, an authentication failure burst inside a sliding window, a template the account has never requested, query parameters the endpoint has never carried, a privileged call within five seconds of a forum view, a supporting authorship link that emits no alert of its own, and a globally rare status code. A correlator unions alerts into an incident when they share an entity, user, address, address owner, object ID or target file, inside a rolling 72 hour window, then resolves attacker and victim roles from the signal basis. Explanations are templates filled from the same fields that appear in the alert payload, so every sentence on screen traces to a field and a line number rather than to a model.

The replay engine merges a thread safe injection queue into the event stream by timestamp, which is how a judge generated attack arrives indistinguishable from real traffic. It streams event, alert, incident and replay state frames over SSE with a monotonic sequence number and a backfill ring for reconnecting clients.

For the red team, Claude returns JSON parameters only: attacker, victim, target, which evasions to apply, timing offsets. It never writes a log line. Deterministic code renders every byte from real benign lines using a response size table measured from the dataset, so a synthetic line cannot be picked out by its length. A critic then rejects any variant whose declared evasions are not genuinely present in the rendered output, which is the check that keeps the per operator metrics from being fiction.

The blue agent proposes rules inside a small domain specific language that is tokenised and parsed by recursive descent into fixed dataclass nodes, capped at depth four and thirty nodes, and walked by a comparator. There is no eval, no exec and no regular expression compiled from a file that a model wrote. Accepted rules append to a YAML file the detector hot reloads by polling mtime between events.

The Elastic layer maps events to an explicit ECS schema and translates the same rule language to ES|QL. Because the rule walker is two valued and ES|QL is three valued, every leaf is made total, so equality becomes a null check plus a comparison and negation inverts correctly instead of silently dropping every null field row.

Challenges we ran into

The brief we were given was wrong in ways we only found by checking. The incident was dated 27 March; it is 13 to 15 March. Several file paths were wrong. We wrote a ground truth document that overrides every other document in the repository, with one rule: where the data and a document disagree, the data wins.

We nearly shipped a false claim. Every source said the attacker planted forum post 1042, and our detector was printing that sentence in its explanation text. Then we checked the object. Post 1042 first appears seven months before the incident, all ten employees read it, and it has been edited 112 times, thirteen of them by the attacker as ordinary use. Posting returns a redirect that never names the object it created, so the log physically cannot show who planted anything. We removed the claim everywhere and replaced it with what the timing does support, that he was at the vector immediately before and after, which is just as damning and survives cross examination. That finding now carries medium confidence and says why on screen.

Our headline number was suspicious. Detection came back at one hundred percent across all two hundred variants, which should worry you rather than please you. We ran a negative control, feeding genuinely benign traffic to the harness labelled as an attack, and confirmed it scored zero. Then we measured which signal was actually doing the work, because nine rows of one hundred percent tells you nothing.

Smaller things bit hard. Our timestamps disagreed with the raw lines they sat beside, because we assumed a daylight saving change the CSE dataset does not contain. Raw log lines wrapped mid token, so usernames rendered broken in the one place that has to look like a file. The judge panel reported success while injecting events that raised nothing, because the renderer emits raw text rather than parsed fields and every signal was reading fields that were not there.

Accomplishments that we're proud of

We attacked our own detector two hundred ways and published the number that made it look worse. Detection is one hundred percent with zero false positives across 22,956 held out events, but one signal, the one that notices an account appearing on a machine it has never used, is the entire margin. Strip it out and detection is 63.5 percent. We say that before anyone asks.

The evasions genuinely worked. Slow password guessing suppresses our burst detection completely, zero of forty six against a control that fires every time. Widening the gap between a victim reading a post and the privileged call suppresses our best signal completely. They were caught anyway because the signals overlap, which is a more interesting result than a clean sweep.

Attacker attribution is 53 percent with zero misattributions. Every miss is the system declining to name anyone rather than naming the wrong person. In an investigation those are not remotely the same failure, and a tool that says it does not know beats one that accuses an innocent employee.

The validation gate rejects. We built a rule that matches the literal string csrf, which appears in the real attacker's payloads. It reproduces the incident perfectly and catches zero of forty held out variants that rename the parameter. It failed, and we kept it on display, because a gate that never rejects anything is not a gate.

Sentry found a real bug rather than a decorative one. Our replay advertised six log hours per wall second and delivered about four, a 62 percent drift, because each deadline was measured from when the previous event actually emitted and a timer rounding error compounded across 481 waits. Anchoring each deadline on the previous one took it to 0.1 percent.

For Elastic we proved the query translation rather than asserting it. Each rule is evaluated twice, once by our own walker over the real events and once by a three valued local executor running the generated query, and the matched line sets are compared. Then we patch in three deliberately broken translators and confirm the agreement disappears, because two evaluators agreeing could otherwise just mean they share a bug.

316 tests pass and the whole pipeline rebuilds from the raw log file with one sequence of commands.

What we learned

Overlapping signals hide which assumption is load bearing. Our per operator table looked like nine perfect scores until we measured which signal was the sole detector on each variant. That one column reframed the entire result and is now the first thing we show.

The environment does more work than the cleverness. Every employee in the CSE dataset uses exactly one machine, so an account appearing anywhere else is arithmetic rather than inference. That is why our strongest signal is strong, and it is a property of this company, not of our design.

Held out evaluation is the only kind that means anything. Once the generator could produce variants from seeds and personas the proposer had never seen, rules that looked excellent started failing honestly, and that is when the numbers became worth quoting.

The thing that most improved the product was removing claims. The heuristic labelled as a heuristic, the dismissed lead stated at its real size, the attribution that declines to guess. Every one of them was a place we had originally overstated, and every one reads as more credible than the confident version.

What's next for Minny

We want to extend the rule language so the blue agent can express windowed counts that translate cleanly to ES|QL, which is the one construct we currently refuse to translate rather than approximate. We also want pull request review comments fed back into the agent, so a rejected rule learns from the human who rejected it.