Mowgli

An AI that shrinks big language models until they run on an old phone — and lets the phone, not the AI, decide whether it worked.

Comment

Mowgli

What it is

Mowgli is a laboratory that runs itself.

You give it a language model and a target — "run this at four words a second on that phone." Then you leave. It forms a hypothesis, builds the thing that would test it, sends it to a real phone over a USB cable, measures what happens there, and decides from the measurement whether the idea was worth keeping.

Then it forms the next hypothesis, knowing what the last one taught it.

It is built the way a laboratory is built: a part that thinks, and a part that measures, and a strict wall between them. The thinking part may propose anything at all. It may not decide whether its own proposal worked. That belongs to the measurement, and the measurement happens on real hardware.

That wall is the whole design. Everything else follows from it.

The two halves

The laboratory — everything here is deterministic

The laboratory is a set of instruments. Each one does exactly one thing, the same way, every time.

There are instruments that make a model smaller — several of them, differing in how much precision they give up. There is an instrument that pushes a model onto a phone. Instruments that time it, that measure how much memory it holds, that watch the device's temperature while it works. And an instrument that compares the words the shrunk model produces against the words the original produced.

None of these instruments reason. They do not have opinions about what should be tried next. They execute a specification and report what happened, including when what happened is a failure.

The laboratory grows by gaining new instruments. It never gets cleverer.

The reasoning — this part is not deterministic, and it is not trusted

Above the laboratory sits the part that decides what to try.

It reads everything that has ever been measured on this model, forms a hypothesis about what might help, and writes a specification the laboratory can execute. It is genuinely uncertain work. Two runs may take different paths.

There are three of these, and they are interchangeable because they all speak through the same narrow opening — a single function that takes everything known so far and returns one proposed action.

The search. It treats the work as a tree. Each version of the model is a node; each transformation is a branch. It extends whichever version currently performs best, and when a branch fails it walks back to the last good version and tries a different direction. This is the default, and it needs no internet connection and no API key at all.

The simple planner. Works through a list. Used to bring up new hardware and as a control when comparing the others.

The agents. A pipeline of ten components, described below. This is where the language models live.

None of the three can accept anything. The strongest statement any of them can make is "I have no hypothesis left worth the phone's time" — and when that coincides with an unmet target, that is a real conclusion in its own right: not achievable with the instruments currently available. That is not a failure message. It is the signal to build a new instrument.

The agents

When Mowgli uses language models to reason, ten components run in two stages.

Five of them are deterministic and run locally. Before any reasoning happens, these establish facts: what model is this, what architecture, how many layers, how many parameters. What machine are we on — cores, memory. Which precisions are even feasible given what is free. What does the model's own documentation say. And a per-layer plan: which layers are safe to compress, which should be preserved, which are risky.

These cost nothing and use no API. Their job is to make sure the expensive stage starts informed rather than guessing.

Five of them reason, and these use a language model. A researcher forms hypotheses from the observation. An optimiser turns a hypothesis into a plan. An executor turns the plan into something runnable. A benchmarker reasons about what the resulting numbers would mean. A synthesiser produces the verdict the loop consumes.

Each agent runs a bounded reasoning loop rather than a single call — it thinks, checks whether it has converged, and thinks again, up to a limit set per role. The limits are hard, because an unbounded reasoning loop against a paid API is a way to lose money quietly.

Every step is written to a ledger. Work already done on a model is remembered in a local database, so the same model seen twice is recognised.

Which language model does the reasoning is a single line in a configuration file — Anthropic, Azure, or Groq. Not a code change.

And if it fails entirely, the system does not. When the API ran out of credit mid-run, the loop fell back to the search and terminated with a valid conclusion. The brain dying does not kill the laboratory. That is what the wall between them is for.

What it remembers, and why it never starts over

This is the part that makes Mowgli a laboratory rather than a script.

Every experiment becomes a permanent record. Not a log line — a structured file that holds what was tested, the full history of transformations that produced it, which device it ran on, what that device's cool-down policy was, every individual timing rather than just the average, whether the words matched the original, and what reference they were compared against.

Failures are recorded exactly as carefully as successes. A rejected candidate produces a complete record. Seven optimisations have been measured and thrown out so far, and those seven records are among the most valuable things in the system.

Before deciding anything, the reasoning reads all of it. Not a summary — the records themselves. It knows which transformations have already been tried on this model, which ones were accepted, which were rejected and why.

And after the measurements, a separate pass reads every record and writes down what held up across them. These are conclusions rather than observations, and they live in a different place so that a conclusion can never overwrite a measurement. Each one is scoped — true of this architecture and this model, never declared as a universal law — and carries a confidence tied to how much evidence it actually rests on. Thin evidence is labelled thin.

Those conclusions feed the next run's reasoning. A technique shown across enough experiments to be useless is not measured again; the phone's time goes to something unknown instead.

So the second run on a model does not begin where the first one began. It begins where the first one ended. Run it a third time and it starts from everything the first two learned. The interface says so out loud — it will tell you how many experiments it remembers about a model before you press anything.

The same applies across models. Conclusions carry a scope, so when a new model arrives, the ones that apply to its architecture come with it. A brand-new model of a familiar family does not arrive at an empty laboratory.

One cycle, end to end

Here is what happens between pressing the button and getting an answer.

It checks whether the model can run at all. It opens the file and reads what tensors are actually inside it, then matches them against what the runtime knows how to consume. Not the model's own description of itself — the actual contents.

That distinction was learned the hard way. One model family carries extra pieces in every attention layer that its own configuration file never mentions. A loader that trusts the description silently ignores three tensors per layer and produces fluent, confident nonsense. No crash. No warning. Just wrong words. Mowgli now reports anything it cannot consume before spending a minute of the phone's life on it.

It establishes the truth. Before shrinking anything, it runs the original at full precision and records exactly what words it produces. That sequence becomes the answer key. Every later version is judged against it.

When the original is too large to run on the phone, the answer key is computed elsewhere — and the record says so explicitly, because the same arithmetic on two different chips does not produce identical results. They agree for about six words and then drift apart. Small, quiet, and it would invalidate a lot of casual comparison if nobody recorded which machine produced the reference.

It forms a hypothesis and builds it. The reasoning proposes one transformation. The laboratory applies it, producing a new version of the model with its full derivation history attached — what was done to it, in what order.

It sends it to the phone and measures it there. Over a cable. On the actual device. With a cool-down before each measurement, because phones throttle when warm and a warm phone quietly reports worse numbers.

Then two gates, in this order, and the order matters.

First: does it still say the same thing? If the words changed, the candidate is rejected immediately and no speed measurement is even taken. A faster wrong answer is not an improvement.

Second: is it actually faster? Measured several times, with the spread recorded. The same optimisation has measured plus thirty-seven percent, plus a hundred and thirty-five, and plus five hundred and thirty-one percent in three runs an hour apart on the same phone. A single number means very little, and Mowgli never reports one alone.

Whatever happens is written down. Then the reasoning reads it and proposes the next thing.

And when a branch dead-ends, it goes back. Every version carries its derivation history, so those histories form a tree. A rejected version is a dead branch; the search returns to the last version that worked and tries a different direction from there. That is not a metaphor — it is literally how the search moves, and it has been observed doing it on the phone.

When the work is too large for the machine you have

Some models cannot be prepared on a laptop at all. Establishing an answer key for a twenty-eight gigabyte model means running it at full precision, which on an ordinary machine takes hours.

For that, Mowgli has a cloud path: it can check a provider's balance, search for the cheapest machine matching what it needs, rent it, wait for it to come up, push the work there, retrieve the results, and destroy the instance — with a spending cap and an automatic teardown so a forgotten machine cannot quietly bill for days.

This has been used successfully. It is not yet wired into the automatic loop — today it is invoked deliberately rather than triggered on its own. Making the loop reach for it by itself, when it recognises that a job exceeds the local machine, is the next step for that piece.

What actually happened — two runs

Both on the same device: a Samsung SM-M305F. Released in 2019. Four gigabytes of memory, of which about one and a half are ever free. Eight Arm cores.

Run one — the small model, measured both ways

The point of this run is that the model fits either way, so it can be measured honestly before and after, on the same phone, with the same starting words.

Before. Mowgli put the original on the phone with everything switched off — full precision, no Arm kernel, no threading, no memory streaming. Nothing it does was active.

513 MB   ·   1.829 words per second

That is the honest starting point.

Then it went to work. Fourteen minutes, unattended.

It confirmed the architecture was one it could run. It reused the answer key it had recorded on an earlier run — the interface showed this as a skipped step, which is the memory working. Then it started proposing.

It tried six different ways of making the model smaller. Five were rejected — every one of them because the shrunk model produced different words from the original. Four-bit compression, twice, in two different forms. Pruning, at two different ratios. And a fuller eight-bit scheme that compresses the embedding table as well — which is interesting, because that exact scheme is accepted on other model families. It fails on this one. That is now on record.

One was accepted: eight-bit weights, at 222 megabytes.

Then it worked on speed, one technique at a time, each measured separately against a fresh baseline:

a hand-written Arm routine                 +163%   kept
re-scoring the final layer                   +9%   kept
preallocating the attention cache          −1.7%   no gain, dropped
reusing scratch memory                     −0.2%   no gain, dropped

The Arm routine is the largest single win in the project. Arm chips have instructions that multiply and accumulate several small integers at once; using them properly, by hand, is worth more than everything else combined.

After.

222 MB   ·   2.503 words per second

Against the original: two point three times smaller, thirty-seven percent faster — and producing word-for-word identical output.

Smaller, faster, and saying exactly the same thing. The last part is what took the work.

Run two — the seven billion model

This is a different kind of result.

Before. Mowgli did not attempt it, and said why:

BEFORE MOWGLI
DOES NOT RUN
28.37 GB · full precision · nothing switched on
The original is 28.37 GB and this phone has 3.6 GB of memory in total.
It cannot be loaded here.

That refusal is deliberate. A phone under that kind of memory pressure does not fail politely — it reboots. Recording "this cannot be loaded here" is a more useful measurement than a restarted device, and it is the honest before-state.

After Mowgli: 8,568,566,911 bytes. And it runs.

That file is still more than twice everything the phone has, and roughly five times what is actually free. It works because of one idea, carried further than usual: instead of loading the model into memory, Mowgli keeps it on storage and reads one layer at a time — then hands those pages straight back to the operating system the moment that layer is finished with. The embedding table too. The output layer too.

Only one piece is ever actually held. About six hundred megabytes, no matter how many layers the model has.

The cost is real and large: every layer is read from storage again for every single word. That works out at roughly thirty-five seconds per word.

Slow. Stated plainly, not dressed up. But slow and the device restarts are different categories, and this phone used to be in the second one.

One thing that came out backwards, pleasantly. The expectation was that handing memory back would cost speed. Measured both ways on a smaller model, with everything else identical:

holding the model in memory      3.52 words per second
handing pages back               4.08 words per second      identical output

Sixteen percent faster, not slower. On a device with barely a gigabyte to spare, releasing memory relieves pressure the whole system was paying for, and a small file returns from the operating system's own cache anyway. Reading it again turned out to be cheaper than holding on. That result was not predicted; it was measured.

What it costs the phone

The seven billion model was left generating continuously for twenty-two minutes, with the device read after every pass:

pass 1     0.0335 words per second     39.9 °C
pass 2     0.0309 words per second     40.0 °C
pass 3     0.0321 words per second     40.6 °C

The speed did not decay. That matters more than it looks — a chip that heats up throttles itself, and speed falling across a long run is the usual signature. It did not happen here. Three passes over twenty-two minutes is enough to see it if it were there.

The chip warmed by seven tenths of a degree. For continuous work on a phone this old, that is close to nothing.

The reason is structural rather than lucky. Because the model streams from storage, the processor spends most of its time waiting for data rather than burning through it. It is a genuine trade: you give up speed and you get back a phone that stays cool. For something running quietly in the background, that may well be the better side of the bargain.

Battery could not be measured, and no figure is claimed. The phone charges through the same cable that carries the commands, so its level rose during the test. Any battery number taken that way would be meaningless. Temperature is the honest signal, and it is the one reported.

What the whole thing proves

Five models have been through this. Two entirely different model families. One runtime, unchanged between them. One old phone.

SmolLM2 135M         513 MB      →   222 MB      2.50 words/sec
Qwen2.5 0.5B         1.84 GB     →   530 MB      4.08 words/sec
SmolLM2 1.7B         6.5 GB      →   2.1 GB      0.13 words/sec    (original reboots the phone)
Qwen2.5 3B           11.8 GB     →   3.3 GB      0.07 words/sec    (original cannot run)
Qwen2.5 7B           28.4 GB     →   7.98 GB     0.03 words/sec    (original cannot run)

But the results are not the interesting part. This is:

A language model, reasoning inside Mowgli, confidently proposed four-bit compression. The phone measured it and rejected it — the words came out different. That rejection went into the permanent record. The language model read it, changed its mind, proposed eight-bit instead, and that passed. Target reached. Nobody touched anything in between.

Then the same gate rejected two of my own ideas. I had worked out on paper that compressing the final layer was twenty-one percent of the arithmetic and an easy win. I built it. Rejected — it changed the words. I built a cleverer version that preserved the words exactly. Measured as no faster at all.

Two of the seven rejections are mine. The gate did not care who proposed them.

That is what the wall is for. Not to make the AI look good, and not to make it look bad — to make sure that when something is claimed, a piece of hardware somewhere already agreed with it.

The AI proposes. The phone decides.

Share this project:

Updates

Submission history