EventBreaker: AI-planned, measurement-backed reliability testing for Java event listeners.
Nemotron-3-Ultra-550B plans the tests, Nemotron-3-Super-120B steers each run, the Nebius sandbox runs it, and code measures the result.
One command finds every listener in an existing Java repo and gets its real classpath. No test code needed.
Nemotron-3-Ultra-550B reads the listener and names the risks in this specific code, each as a fixed test format, never as Java.
Nemotron-3-Super-120B sets fake values so the run reaches the risky call; the real compiled listener runs in the Nebius sandbox.
A real bug in Mifos Payment Hub: a redelivered Kafka message starts the same payment twice (2 calls vs 1 in the baseline).
Tested on 61 listener methods from 9 open-source projects: 45 reliability risks found, confirmed by reading the source before the runs.

Inspiration

Event-driven systems fail quietly. Message brokers like Kafka and RabbitMQ deliver events at least once and sometimes out of order, while the databases, caches and services a listener calls can fail or slow down at any moment. Each listener is a small piece of code that has to stay correct through all of that, and the ways it goes wrong are rarely dramatic: a payment is started twice, a failure is caught and logged while the message is still acknowledged, half of the work is saved and the other half lost. Nothing crashes, so these bugs are found in production, usually by a customer. We wanted a tool that reads each listener, works out what could go wrong in that specific code, and then proves it by running it, without anyone writing a test.

What it does

EventBreaker:

finds the listeners: Kafka, RabbitMQ, JMS, Spring events, Guava / Kill Bill bus
lets Nemotron decide what could go wrong: Nemotron Ultra reads the listener and names the risks it sees in that specific code
turns each risk into a test from a small set of building blocks: deliver the event once, twice, or twice at the same time; make any dependency call throw, throw once, or slow down; check the call order or whether a failure was swallowed. A risk that cannot be expressed this way is reported as not tested, never guessed
runs the listener's real compiled class on the project's real classpath inside a Nebius sandbox; only its dependencies are faked, so its own catches, retries and acknowledgements are what get tested
reports only what it measured: every scenario is compared with a fault-free baseline run, and a finding needs a real difference
writes a Markdown report with severity, evidence and a suggested fix.

Tested on 61 listener methods from 9 open-source projects, it found 45 reliability risks that we had confirmed by reading the source code before the runs.

How we built it

Understand : a Java static analyzer (JavaParser) lists the listener's dependencies and calls; a resolver gets the real classpath from Maven or Gradle.
Plan: Nemotron Ultra 550B reads the listener and fills a fixed TestConfig per risk: a call pattern (DUPLICATE / SINGLE / CONCURRENT) and faults (THROW / THROW_ONCE / DELAY) on named dependency methods. It never writes code. Nemotron Super 120B picks fake return values so the run reaches the risky line.
Execute: Python turns the TestConfig into a Java harness (Mockito fakes), compiles it against the real jars, trims it to one JAR, and runs baseline + scenario in the Nebius sandbox (ConTree SDK).
Diagnose: Nemotron-3-Super-120B labels each dependency call (read / write / side effect; money, security, data etc.) by majority vote, a fixed rubric in code sets the severity, Nemotron-3-Super-120B writes the explanation and fix, and Nemotron-3-Ultra-550B reviews the result.

Challenges we ran into

Model-written tests did not compile. Our first version asked the model to write the Java test code. Without seeing the whole project, it guessed class and method names, and patching each broken test did not scale. so the fix we applied is that the model now only fills a small fixed test format and our code generates the Java, so every test compiles.
Tests stopped before the risky code. Real listeners check values, cast objects and read configuration before doing any real work, so with plain fakes many runs ended early and tested nothing. To ensure we reach the risky path a model picks fake return values that open the right path, retries once with feedback on where the run stopped, and reuses setups that worked. This is still the main limit on new projects.
Same evidence, different verdicts. When a model judged severity directly, the answer changed between runs, even at temperature 0 so in the current version models answer only small, closed questions by majority vote, and fixed rules in code decide the severity.
Real projects are hard to build and run. Some needed build files that were never published, one had a setting that silently emptied the classpath, and some targeted newer Java versions; large dependency sets timed out when uploaded to the sandbox so we pinned release builds, a fallback for filtered build settings, one trimmed JAR per test, and a sandbox JDK chosen from the compiled code.

Accomplishments that we're proud of

Real bugs in real products: a duplicate payment start in Mifos Payment Hub, lost work behind an acknowledged Kafka message in Apache Fineract, a lost audit record in DIGIT, and a login-lockout race in Fineract.
No finding without a measurement, and no generated test that fails to compile.
Works on an existing repository with zero setup: the classpath comes from Maven or Gradle automatically, and it installs with one pipx command.
Cheap to run: about 2,800 model calls and roughly $8 for the whole development and evaluation; the final sandbox run on 22 listeners cost $0.20.

What we learned

Give the model a form, not a blank page. Nemotron is very good at deciding what to test and weak at writing a correct harness for code it cannot see. A fixed vocabulary made the plan reliable and every test readable.
Let code decide, let models answer small questions. Temperature 0 is not determinism on batched servers, and votes at temperature 0 are not independent.
Pick the model per job. In a side-by-side path-setup test, Nemotron-3-Ultra-550B ran out of tokens on one case and missed the key string on the other; Nemotron-3-Super-120B got both right. Nemotron-3-Super-120B now handles the many small calls; Nemotron-3-Ultra-550B plans and reviews.
Reasoning models need defensive parsing. The answer sometimes sits in reasoning_content with an empty content; reasoning text can contain brackets before the real JSON; once, a reply used all 8,192 tokens thinking and gave no answer. Bad replies are re-asked and never cached.
Measure against a baseline. Our first diagnosis flagged "called more than once", which is wrong for code that fans out; comparing with a fault-free run removed a whole class of false alarms.

What's next for EventBreaker

Confirm findings against a real broker and database (Testcontainers), not only fakes.
Bugs that span several methods of one class, and failing only the n-th call of a method.
Evaluate the listener types it already recognizes (@SqsListener, @StreamListener).
A GitHub Action that runs on pull requests that touch a listener.
Report the findings to the maintainers of the projects we tested.