Vifu: Mobile AI Optimization Workbench

Build, iterate and optimize Mobile AI Apps with on-device models.

Comment

Inspiration

Running a model is only one part of an Agent App. Developers still need to connect it to product code, manage agents and devices, trace failures, and measure whether an optimization improves the real application.

A standalone benchmark can show tokens per second. It cannot show which device, Provider, model, or inference stage produced a slow or incorrect application result. Small app and game teams should not need to build a server, database, device protocol, TUI, and trace viewer before they can debug local AI.

What Vifu does

Vifu is an open-source development platform for Agent Apps that use on-device models. One local Vifu binary provides a Server, live TUI, Dashboard, OpenAI-compatible API, and SQLite trace store.

An App can contain multiple devices, Providers, Agents, and models. A phone, game, computer, or embedded system joins as a Gateway. Vifu then connects every request to the device and model that ran it.

Each Trace can show:

  • the App, device, Agent, Provider, and model.
  • total latency, time to first token, and output rate.
  • Queue, Load, Tokenize, Prefill, First token, Decode, and Validate stages.
  • bounded errors and optional application-approved input and output.

Developers can also call configured device Agents through one endpoint. This makes a phone or embedded runtime part of the same development workflow as a desktop Agent App.

Mobile Arm demonstration

The public release includes two Android applications based on Arm's official examples/llama.android sample:

  • Vifu Starter Optimized uses Vifu's Arm-optimized llama.cpp backend.
  • Vifu Starter Baseline uses the compatibility llama.cpp backend.

Both APKs can run the same GGUF on one phone. Each APK pairs with the same Vifu App and appears as a separate device and Agent. A developer can enter the same message, watch both runs in the TUI, and inspect their model stages in the Dashboard.

The optimized path was verified on a Xiaomi Android 16 ARM64 phone with Qwen2.5 0.5B Q4_K_M. One recorded run produced an 847 ms time to first token and 17.1 output tokens per second. Vifu stores the device, backend, model, and stage data so the developer can repeat the measurement on the target hardware.

Developer experience

Vifu provides two direct Arm entry points:

  • Desktop Arm: one uv command starts a Python Web Research Agent App, local Server, Dashboard, Foundry Local model, and tracing workflow on macOS.
  • Mobile Arm: download Vifu and an Android APK, select a GGUF, pair from Dashboard Overview, then inspect the phone's inference performance.

Product integrations use the same Runtime and Gateway model across Python, TypeScript, Android, iOS, Godot, and Rust. The complete SDK tutorials, release downloads, architecture, and source-build instructions are in the public GitHub repository.

How we built it

  • Rust provides the Runtime, Gateway, Server, TUI, routing, and persistence.
  • Kotlin and modular AARs embed the optimized and baseline llama.cpp Providers.
  • React and TypeScript provide the embedded Dashboard and Trace Explorer.
  • SQLite stores Apps, devices, Agent configuration, and trace history locally.
  • UniFFI and WebAssembly provide Python and TypeScript integration paths.

What we learned

Model speed has little meaning without attribution. An optimization is useful only when the correct model ran on the intended device and the complete Agent App produced a valid result. Vifu makes that evidence visible in the product's real runtime instead of a separate benchmark harness.

What's next

Next we will publish repeated optimized-versus-baseline measurements across more Arm phones and models, add more on-device framework adapters, and extend the same trace and deployment workflow to more Agent Apps and games.

Built With

Share this project:

Updates

Submission history