DelegateDoctor

DelegateDoctor finds the ExecuTorch/XNNPACK fallback actually slowing your Arm model, applies proven repairs, and keeps them only if a real Android phone proves they are correct and faster.

Comment

Inspiration

DelegateDoctor was built for the Mobile AI track of the Arm Create: AI Optimization Challenge. It targets local inference on Arm-powered Android devices using PyTorch, ExecuTorch, and XNNPACK.

Getting a PyTorch model to run with ExecuTorch is only part of deploying it efficiently on mobile. A model can successfully lower to ExecuTorch while some operators still fall back outside the XNNPACK delegate. Those fallbacks can quietly become major performance bottlenecks.

The problem is that operator counts alone do not tell you what actually matters. A model may contain many fallback operators that barely affect latency, while one fallback pattern can dominate runtime.

I built DelegateDoctor around a simple idea:

Not all fallbacks are equal. Optimize runtime, not operator counts.

I wanted a tool that could go beyond saying "this operator did not delegate." DelegateDoctor measures where the model is actually spending time on an Arm device, identifies repairable fallback patterns, verifies that a repair preserves correctness, and finally asks the most important question: did it actually make the model faster on the target device?

I also wanted DelegateDoctor to be more than a collection of fixes I discovered myself. It is open source and designed to grow through community contributions. When a developer discovers a new ExecuTorch/XNNPACK fallback pattern and finds a safe solution, they can contribute it as a new validated DD repair rule. The goal is to build a shared catalog of tested solutions so that developers do not have to repeatedly rediscover the same backend problems and workarounds.

Find a fallback. Validate a repair. Contribute the rule. Everyone benefits.

What it does

A developer points DelegateDoctor at a PyTorch model:

delegate-doctor optimize model.py

DelegateDoctor then:

  1. Exports the model using torch.export.
  2. Lowers it to ExecuTorch with the XNNPACK backend.
  3. Profiles the original model on an Arm64 Android target.
  4. Ranks non-delegated work by measured runtime, rather than operator count.
  5. Matches known fallback patterns against a deterministic repair catalog.
  6. Applies the repair across validated matching sites.
  7. Verifies the repaired model on the host.
  8. Verifies it again on the Android target.
  9. Checks host/device backend fidelity.
  10. Benchmarks the original and repaired programs.
  11. Keeps the repair only when it is correct and actually faster on the device.
  12. Produces an optimized .pte model and a self-contained HTML report.

The philosophy behind the pipeline is:

The repair proposes a change. DelegateDoctor verifies it. The Arm target decides.

DelegateDoctor currently contains three deterministic repair rules for real ExecuTorch/XNNPACK fallback patterns:

  • DD-001: rewrites non-last-dimension softmax into an equivalent XNNPACK-friendly layout.
  • DD-002: removes redundant alias operations that can fragment delegation.
  • DD-003: rewrites an unsupported avg_pool2d padding configuration into an equivalent form that XNNPACK can delegate.

It also recognizes when a model is already fully delegated or when a fallback exists but no known repair is available. DelegateDoctor does not manufacture an optimization just to produce an output model.

How I built it

DelegateDoctor is written in Python and integrates several parts of the PyTorch mobile deployment stack:

  • PyTorch and torch.export
  • ExecuTorch 1.4
  • XNNPACK
  • ExecuTorch event tracing and ETDump
  • Android Debug Bridge
  • native Arm64 ExecuTorch runners
  • host and device numerical verification
  • runtime benchmarking
  • structured JSON results
  • self-contained HTML reporting

A major design decision was to make the core optimization path deterministic and evidence-driven.

Each repair rule has explicit preconditions and a defined graph transformation. A candidate repair cannot simply increase delegation and call itself successful. It must pass several independent gates:

  1. Host correctness — original host output vs. repaired host output.
  2. Device correctness — original device output vs. repaired device output.
  3. Backend fidelity — checks that lowering to the device backend has not introduced unacceptable numerical differences.
  4. Device benchmark — the repair must actually improve measured performance before DelegateDoctor accepts it.

For profiling, I did not want to use delegated operator percentage alone. DelegateDoctor instead derives runtime-weighted delegation from ExecuTorch traces. This measures how much of the observed execution time is actually running through delegated work.

Physical arm64-v8a Android hardware is the supported and validated benchmark target. DelegateDoctor can also use an already-running Arm64 Android emulator as a best-effort fallback, but it does not create, provision, or validate emulator environments, and emulator latency is clearly labeled as non-physical evidence.

DelegateDoctor also includes an experimental opt-in AI path for exploring unknown patterns, but all official benchmark results were produced by the deterministic repair catalog without AI repair.

Challenges I ran into

Measuring the right thing

At first, it was tempting to judge optimization quality by looking at how many operators XNNPACK delegated.

That turned out to be misleading.

A small fallback can consume a significant portion of runtime, while many fallback operators can have almost no practical performance impact. This led me to make runtime-weighted delegation one of DelegateDoctor's core metrics.

Instead of asking:

"How many operators delegated?"

DelegateDoctor asks:

"How much measured execution time is actually delegated?"

Correctness across backends

A graph rewrite can be mathematically reasonable in PyTorch and still behave differently after lowering to another backend.

Because of that, I did not want correctness checking to stop at eager PyTorch. DelegateDoctor separately validates host semantics, device semantics, and backend fidelity before using performance as the final decision.

This became especially important while developing DD-001. An obvious rank-4 layout transformation delegated successfully but produced incorrect results through the actual XNNPACK path. That forced me to redesign the repair rather than simply accepting the graph because it delegated.

More delegation does not automatically mean faster

Another important challenge was resisting the assumption that a higher delegation percentage must mean a better model.

DelegateDoctor therefore does not accept a repair because the graph looks cleaner or because delegation increased. It benchmarks the candidate on the target.

This is a key part of the design: performance is measured, not assumed.

Real mobile benchmarking is messy

DD-002 exposed one of the most interesting challenges in the project.

GhostNet showed a strongly bimodal baseline latency distribution on the physical test phone. An early 20-sample run looked extremely stable, but all of its samples had landed in the same execution mode. Low variance within that run therefore did not mean the measurement was representative.

I repeated DD-002 using five interleaved repetitions. Two independent validations produced:

Run Before p50 After p50 Speedup
Validation 1 721.71 ms 499.19 ms 1.45x
Validation 2 737.06 ms 519.27 ms 1.42x

Both passed host correctness, device correctness, and backend fidelity.

That experience taught me an important lesson: a benchmark can be internally consistent without being representative.

Accomplishments I'm proud of

The final validation was performed on a physical RMX2030 Android phone using arm64-v8a, Android 10, and a Qualcomm Snapdragon 665.

DD-001 — non-last-dimension softmax

DD-001 was tested across seven models, and all seven repairs were accepted.

Model Runtime delegation p50 before → after Speedup
PSPNet 53.0% → 99.6% 224.57 → 73.26 ms 3.07x
DeepLabV3+ 59.1% → 100.0% 376.65 → 132.67 ms 2.84x
SmallNet 95.4% → 100.0% 4.63 → 2.42 ms 1.91x
Linknet 58.3% → 100.0% 258.99 → 163.40 ms 1.58x
FPN 70.8% → 93.4% 386.79 → 253.26 ms 1.53x
U-Net 73.5% → 93.7% 525.55 → 343.40 ms 1.53x
U-Net++ 79.5% → 93.4% 625.03 → 424.54 ms 1.47x

DD-002 — redundant alias operations

GhostNet contained 32 matching alias sites.

After resolving the bimodal benchmark behavior with repeated interleaved measurements, DD-002 was accepted at approximately 1.4x faster, while passing all correctness and fidelity checks.

DD-003 — avg_pool2d padding

DD-003 was validated on two different architectures:

  • Inception V3: runtime delegation 53.3% → 100.0%, 1.95x faster
  • DenseNet169: runtime delegation 95.5% → 99.9%, 1.06x faster

The DD-003 transformation was also verified to be bit-exact in eager PyTorch before device benchmarking.

Healthy-model controls

I also tested models where DelegateDoctor should not perform a repair:

  • MobileNetV2: 100% runtime delegation → NO_REPAIR_REQUIRED
  • ResNeXt50-32x4d: 100% runtime delegation → NO_REPAIR_REQUIRED
  • ConvNeXt Small: fallback detected, but no known safe repair → NO_REPAIR_AVAILABLE

I am particularly proud that DelegateDoctor can say "do nothing."

An optimization tool should not modify a model simply because the user ran it.

The project also has more than 1,200 automated tests, covering graph analysis, repair rules, correctness verification, Android target handling, reporting, security boundaries, CLI behavior, and other parts of the pipeline.

What I learned

The biggest lesson from building DelegateDoctor is that mobile AI optimization is not simply about maximizing delegated operator count.

The questions that actually matter are:

  • Which fallback is consuming meaningful runtime?
  • Is there a safe transformation for it?
  • Does the transformation preserve the original model's behavior?
  • Does it remain correct after backend lowering?
  • Does the real Arm device actually become faster?

I also learned how different layers of the PyTorch mobile stack interact: torch.export, ExecuTorch lowering, XNNPACK partitioning, event tracing, Android execution, and numerical verification.

Most importantly, I learned how easy it is for benchmarking to create false confidence. Real devices have scheduler effects, thermal effects, different execution modes, and noise that synthetic graph metrics cannot capture.

That is why DelegateDoctor ultimately treats the target device as the final authority.

A theoretically better graph is not enough.

The optimization has to survive the real hardware.

What's next for DelegateDoctor

The three rules in the current catalog are a proof of the larger idea: fallback knowledge can be turned into reusable, validated repairs instead of being rediscovered independently by every mobile AI developer.

The next step is to expand the deterministic repair catalog as more real ExecuTorch/XNNPACK fallback patterns are discovered and validated.

I would also like to improve benchmark robustness, add richer explanations for why particular graph regions failed to delegate, and validate DelegateDoctor across more Arm Android hardware and newer devices.

The experimental AI exploration path could eventually help propose candidate transformations for previously unseen patterns, but I want DelegateDoctor to keep the same safety principle:

An AI suggestion is never an optimization by itself. It still has to pass deterministic correctness checks and prove itself on the target device.

My goal is for DelegateDoctor to become the tool a developer reaches for when an ExecuTorch model runs on Arm, but not as fast as expected.

Instead of manually searching through graphs and traces, the developer should be able to ask:

"What is actually slowing my model down, can it be repaired safely, and does the fix really help?"

That is the question DelegateDoctor is built to answer.

Challenge-period development

DelegateDoctor was created and meaningfully developed during the Arm Create: AI Optimization Challenge period. The submitted implementation, repair catalog, Android execution pipeline, correctness verification, benchmarking workflow, reporting, examples, tests, and physical-device validation were developed as part of this challenge.

Built With

Share this project:

Updates

Submission history