ThermalPilot

On-device LLM inference scheduler for Android that prevents thermal throttling by dynamically hot-swapping quantization (INT4↔INT8), thread count & context length — keeping tokens/sec stable.

Comment

ThermalPilot — Project Story

Inspiration

Every developer who has tried running an LLM on a phone has seen the same thing: great performance for the first two minutes, then a silent collapse. Token generation that starts at 15 tokens/sec quietly falls to 3 — not because of a crash or an error, but because the SoC quietly raised its temperature above the throttling threshold and the OS cut the CPU frequency in half.

We were benchmarking on-device inference during a late-night session and noticed that the model wasn't getting slower — it was getting thermally murdered. The phone was hot, the tokens/sec chart looked like a cliff, and there was absolutely nothing the inference engine was doing about it. No adaptation, no warning, just silent degradation.

That's the problem ThermalPilot was built to solve: make on-device LLM inference thermally aware.

What it does

ThermalPilot is an Android app that wraps llama.cpp on-device inference inside a real-time thermal-and-battery-aware scheduling loop. Instead of running the model flat-out until the SoC throttles it, ThermalPilot reads the device's thermal sensors every 2 seconds and proactively adapts:

  • Hot-swaps quantization between INT8 (high quality, more heat) and INT4 (lower memory, less heat) without stopping inference
  • Scales thread count from all big cores down to a single thread as thermal pressure rises
  • Shrinks the context window to reduce memory bandwidth and heat
  • Keeps tokens/sec stable across the full session instead of letting performance collapse

The app has two modes: a 20-minute benchmark session with a live telemetry dashboard (TPS chart, SoC temperature, battery temperature, policy state), and an interactive on-device chat interface where you can talk to the model directly. A built-in model downloader lets you paste any HuggingFace .gguf URL and download directly to the device with auto-resume on dropped connections. The whole thing runs completely offline — no cloud, no API keys.

How we built it

The stack is Flutter + Kotlin + llama.cpp.

Flutter (Dart) handles the UI layer: the home screen, live fl_chart dashboard, chat interface with streaming token bubbles, and the summary/export screen. The thermal scheduler FSM and inference engine wrapper also live here.

Kotlin (MainActivity.kt) bridges the gap to Android's sensor APIs via a MethodChannel:

  • PowerManager.addThermalStatusListener (API 29+) for the official thermal status integer
  • /sys/class/thermal/thermal_zone*/temp sysfs parsing as a fallback on older devices
  • BatteryManager ACTION_BATTERY_CHANGED intent for battery level and temperature
  • /sys/devices/system/cpu/*/cpufreq/scaling_max_freq for CPU topology (big vs LITTLE core detection)

llama_cpp_dart 0.9.0-dev provides the Flutter FFI binding to llama.cpp. This version ships pre-built AARs — libllama.so, libmtmd.so, and libggml*.so — meaning no NDK build step is required. The engine runs inference in a dedicated Dart isolate so the UI thread never blocks.

The scheduler is a hysteresis FSM with four states (COOL → WARM → HOT → CRITICAL), requiring 2 consecutive hotter readings to downgrade performance and 3 consecutive cooler readings to upgrade — preventing rapid oscillation and protecting the model from constant reloads.

State Threads Quant Context
COOL 4 (Big cores) INT8 4096
WARM 3 INT8 2048
HOT 2 (Little cores) INT4 1024
CRITICAL 1 INT4 512

Challenges we ran into

The libmtmd.so crash. The biggest blocker was a native library linking failure on startup: dlopen failed: library "libmtmd.so" not found. This turned out to be a fundamental issue with llama_cpp_dart 0.2.x — the package was published to pub.dev without its llama.cpp git submodule, so the CMake build couldn't find the multimodal source files. The fix was migrating to 0.9.0-dev.12, which ships pre-built AARs that bundle all required .so files.

Thermal API fragmentation. Android's thermal APIs are notoriously inconsistent across OEMs. The PowerManager API (API 29+) is clean but many devices always report status 0 regardless of actual temperature. The sysfs fallback works on older devices but is locked down on Android 10+ for most apps. We implemented both paths with graceful fallback and cross-reference them to get the most accurate reading possible.

Download reliability. HuggingFace .gguf files for even a 0.5B model are 400–530 MB. On mobile networks, connections drop mid-download constantly. The first version of our downloader would restart from zero on every failure. We rebuilt it with HTTP Range header support for resume, automatic retry with exponential backoff, and partial file preservation so a restarted download picks up exactly where it left off.

Model hot-swapping without crashes. Switching quantization mid-session requires tearing down the entire llama.cpp engine, releasing the isolate, spawning a new one with a different model file, and reconnecting the chat session — all while the benchmark loop is still trying to run. Getting this teardown/respawn cycle right without deadlocks or null pointer exceptions took significant iteration.

Accomplishments that we're proud of

  • Zero-internet on-device inference with a model downloaded directly from HuggingFace into app storage
  • A thermal FSM that actually holds TPS stable across a 20-minute session while a baseline (no adaptation) collapses in the second half
  • A full streaming chat UI running against a local llama.cpp model with proper context shift, so the conversation window never overflows
  • CPU topology detection that correctly identifies big vs LITTLE cores on arm64 devices and uses that to set thread affinity intelligently
  • Clean CSV export of the full telemetry session (TPS, SoC temp, battery temp, policy state, thread count, quant tier, context length) for offline analysis
  • A live telemetry drawer accessible during chat showing real-time SoC temperature, battery level/temp, CPU core layout, and top thermal zones

What we learned

On-device AI is not just a "run the model" problem — it's an embedded systems problem. Thermal management, memory bandwidth, SoC frequency governors, and battery chemistry all interact in ways that pure software benchmarks completely ignore. The gap between peak performance (first 2 minutes, CPU cold) and sustained performance (20 minutes, SoC at 45°C) can easily be 4–5×, and no existing Flutter/llama.cpp integration we found addressed this at all.

We also learned that the Android thermal API ecosystem is still maturing. PowerManager.getThermalHeadroom() and addThermalStatusListener are the right APIs, but OEM implementation quality varies enormously — some devices never report above status 0, some report status 4 at 38°C. Building a robust scheduler means treating these as inputs to a voting system, not ground truth.

What's next for ThermalPilot

  • 🔭 Speculative decoding — use the INT4 model as a draft model and INT8 as the verifier for free quality improvement while staying thermally efficient
  • ⚡ Hexagon NPU support — the llama_cpp_dart 0.9.0 Hexagon AAR enables inference on Snapdragon's DSP, which generates far less heat than the CPU cores
  • 🎛️ Adaptive sampler tuning — reduce temperature and top-p during HOT/CRITICAL states to generate shorter, faster responses and avoid context overflow
  • 🔌 Widget & background service — expose a persistent background inference service so other apps can query ThermalPilot's model via a local API
  • 🧠 Multi-model routing — load a tiny 0.5B model for CRITICAL state and a 3B model for COOL state, routing prompts to whichever fits the current thermal budget

Built With

Share this project:

Updates