LogicScout

LogicScout meets an unfamiliar system, runs experiments to learn its hidden rules, compiles a controller that needs no model, and notices and adapts when the rules quietly change.

Comment

Inspiration

The world quietly runs on machines that nobody fully understands anymore, factory equipment whose manual vanished years ago, interfaces a whole business depends on that were never documented, plants that behave differently after every maintenance window, and satellites that keep working far beyond the reach of anyone who could open them up. The people who keep such systems alive learn them the slow way, by poking at them, forming hypotheses, testing them and writing down what holds, and we wanted to build an AI that does exactly that work in the open, so that its knowledge is something you can read, run, check and trust rather than something you have to take on faith.

What it does

LogicScout is handed a system it has never seen, with nothing but the names of its controls, the names of its readings and a goal, and it turns that ignorance into working competence in five moves that each leave inspectable evidence behind. It first reads public documentation about this kind of system through Tavily and turns cited passages into prior beliefs that still have to earn their place. It then lets NVIDIA Nemotron propose rival explanations for everything it observes, while code chooses the experiment on which those rivals disagree most and runs it against the real system, so that every explanation is supported or refuted by an exact observation such as "predicted 9 in the queue, saw 8 at step 7". What survives becomes an executable theory written in a small rule language that a deterministic predictor runs step by step, and in the recorded run on the maritime relay LogicScout proposed 158 explanations, ruled out 111 by evidence, and recovered on its own that the buoy only delivers in a short window every six steps and never more than two messages at a time. That theory is then compiled into a controller that operates the system without calling the model at all, and before the controller is trusted with control it has to pass a reality trial on a fresh copy of the real system, because a plan that only works inside its own model is worth nothing. Finally, when we change the hidden rules without telling it, its predictions break, a drift monitor confirms that trusted rules are failing rather than raising an alarm on noise, and LogicScout goes back to experimenting on exactly the rules that broke, rebuilds its controller and reaches the goal again, which in the recorded run took 23 steps to notice and 22 more to recover.

Around that loop sits everything a skeptical operator would ask for. An independent referee, written as code the scout can never import, scores every controller on situations it never saw and audits every prompt for leaked hidden rules, and in the recorded relay run it measured success on half of the unseen cases while random and idle play never succeeded once. A what if console lets you ask the learned theory what a sequence of actions will do and then test that prediction against the real system with a check or a cross per reading. One click exports the operating manual LogicScout wrote for itself and a standalone controller in plain Python. And because anyone could suspect a hand picked demo, the Unknown system option generates a brand new machine for every visitor, so the system you watch it learn has never existed before.

How we built it

Reasoning runs on NVIDIA Nemotron through Nebius Token Factory, with nvidia/Nemotron-3-Ultra-550b-a55b proposing rival hypotheses, revising theories and synthesizing controllers, and nvidia/nemotron-3-super-120b-a12b designing routine experiments and turning Tavily sources into cited, testable claims, all through structured output with schema checks, automatic escalation from Super to Ultra and a log entry for every call carrying the model id the API actually returned. The architecture is deliberately built so that the model proposes and code decides, which means every belief the model contributes is parsed into a whitelisted rule language, simulated by a deterministic predictor and judged against real experiments, and a revision is only accepted when it improves accuracy on experience it was not fitted to. A code side rule inducer and a data driven rule miner add their own rivals to the same contest, a novelty planner keeps experimenting even when every rival agrees so that a mechanism nobody proposed can still be discovered, and a robust planner chooses actions that work under every theory that is still plausible rather than gambling on the favourite. Six worlds sit behind one isolation contract that exposes only four calls, manual, reset, observe and act, namely a maritime relay, a thermal rover, a satellite payload, a water treatment plant, a real HTTP device service running as its own process, and Forge, a generator that composes delays, cooldowns, windows, capacities, lockouts and hidden modes into systems nobody has seen. The product is a FastAPI service streaming every event over Server Sent Events into a React and TypeScript interface, with an append only logbook whose digest chain is anchored in the public git history, 266 automated tests, continuous integration on every push and a live deployment on Render.

Challenges we ran into

The hardest challenge was making the evidence impossible to dismiss, because a system like this can easily look smarter than it is. Prediction accuracy flatters any model of a system where most readings rarely change, so we added accuracy on readings that actually change and accuracy several steps ahead, where the naive baseline scores zero by construction. Validating a controller only inside the model that produced it is circular, so we added the reality trial on a fresh copy of the real system and feed every failed trial back as new experience. A drift detector that fires on an incomplete theory is useless, so we rebuilt it around rules that have earned trust during control, which raised detection of abrupt changes from one in ten to eight in ten and gradual changes from one in twelve to twelve in thirteen while cutting false alarms to a quarter. Reasoning models also spend their output budget on thinking, which at first left most deep calls without an answer, so we measured the behaviour precisely, switched structured steps to reasoning off with schema constrained output, and calls went from about 25 seconds and invalid to about 5 seconds and valid.

Accomplishments that we're proud of

We are proud that every claim LogicScout makes can be traced to a raw observation, and that the evidence holds at scale. Across 100 freshly generated systems and 600 held out cases the learned theory and planner reached the goal on 20 percent of cases with a 95 percent confidence interval of 17 to 24 percent, where a classical table model built from the same budget of experience reached 0 percent and random play 3 percent, and on the relay the learned theory predicts 88 to 91 percent of transitions against 76 percent for the nothing changes baseline. We are proud that the full loop from ignorance to control to detected change to recovery has been recorded live on Nemotron and can be replayed by anyone, with the referee, the hidden rules and every model call open to inspection. And we are proud that its compiled controllers keep working even when the model is unreachable, which is exactly the property an operator needs before trusting an AI with a machine.

Design principles

LogicScout is built around a few deliberate principles, and each one is what makes the rest trustworthy. LogicScout never lets the model act on the system step by step, since a controller that has to ask a language model before every action cannot be audited, reproduced or run when the model is unreachable, so the model's job ends where the compiled controller begins. It never treats public documentation as fact, because manuals are often wrong, which is precisely why someone needs LogicScout in the first place, so every Tavily claim stays a hypothesis until the system itself confirms or contradicts it. It never sees the hidden rules, the referee's cases or the evaluation code, and that isolation is enforced by the code structure, by tests and by an audit of every prompt, because a discovery made with a peek is not a discovery. It never claims to know the true inner mechanism of a machine, only rules that are behaviourally equivalent to it within everything it has tested, and it keeps rival explanations that fit equally well marked as unresolved instead of quietly picking one. It never deploys a controller that has only been tested in its own imagination, and it never acts outside the operator's declared safety limits. And it does not hide its learning inside an opaque neural model of the system, because every belief it holds is a short rule a person can read, question and override.

What we learned

We learned that a large model is far more valuable as a scientist writing small falsifiable rules than as an operator deciding every step, that public knowledge is a superb starting point and a poor final authority, that reasoning models need explicit control over their output budget before they can be trusted with structured work, and that change detection has to rest on evidence from rules that have earned trust rather than on timers or raw error counts.

What's next

Next we are scaling the model enabled evaluation across many more generated systems, carrying mechanisms learned on one machine into structurally different ones as priors to learn them faster, and connecting real systems through the same four call port contract, starting with device interfaces, test benches and industrial simulators, so that the next unfamiliar machine someone inherits comes with an AI that can learn it, run it and tell them the moment it changes.

Built With

Share this project:

Updates

Submission history