Literally the Smallest 0.5B Param Functional LLM ever built

• 

  Performix Capture Tool

• 

  Draw to the LLM

• 

  And it Responds!

• 

  Or Type to The LLM

Abstract

For the Mobile AI track: we demonstrate an optimized 0.5B parameter LLM that can in theory, and practice, run on a wearable device with a reasonable interactive speed. We optimize this using key insights from Arm Performix to fine tune our custom inference framework for ARM NEON based SIMD using code written in pure Zig, with no package dependencies whatsoever. In order to iterate faster on Performix insights we build our own CLI to parse exported .zipped APX reports and provide symbolic insights directly with line numbers. To improve UX on Wearables we also provide a touch based interface for prompting the LLM. We provide a WatchOS app for Mood Uplifting as a use case in order to demonstrate our LLM framework to end users. We also provide an ELF static executable for demonstrating the PII scrubbing use case. Finally, we provide Linux frameworks and executables in order to benchmark on GCP servers using ARM and provide an alternative use-case in PII masking for log messages on server, and for Android.

What it does

Pixy is a 0.5B parameter class Large Language Model based on Falcon H1 0.5B that we have optimized to run on ~3 tokens/second on an aarch64 deployment target of an Apple Watch (demonstrated on Apple Watch Series 9, but can even run on Apple Watch SE 3 and higher). This is a functional model, i.e. you can actually chat with it and generate intelligible replies. To our knowledge this is the first publicly available and truly working multi-turn conversation LLM running on wearable hardware.

Prior attempts have either been incomplete or resulted in garbage outputs or are yet unreleased (as of August 10th,2026 time of writing):

• Runs a 0.6B param model but provides garbage results on watch
• Incomplete attempt at deploying a 100M param model
• WatchOS 27 - Apple's official Foundation framework will include support

NOTE: Apple is unclear on whether the Foundation framework will route to internet servers or process locally. It is most likely a combination of both.

We not only provide a fully functional LLM albeit with knowledge limitations appropriate to 0.5B params, but also provide a touch gesture based ConvNet trained on our own dataset of 9 emoji to communicate within the tiny screen of a watch wearable.

We also provide a finetuned version of the LLM to operate on the lowest tier of GCP ARM based servers (N4A instance) and demonstrate PII masking as a piped text operator on it. We also do Performix benchmarks on the server in order to demonstrate veracity.

We have tried to keep this writeup as non-technical as possible, even when unavoidable we have only provided high level summaries, for a full AI summarized technical list of our optimizations generated from our code please view optimizations.md in our repository.

Constraints:

• 100 MB peak RAM for a 0.5B param model is the primary constraint on any small device
• Dual Core CPU and no GPU acceleration require extreme matmul optimization
• Restrained interaction space of wearables needs workarounds
• No streaming text provider out of the box

Functional limitations:

• No Tool Calling or Reasoning, those need >=3B params to function well
• The model has limited knowledge, but is fairly accurate
• Limited context window due to constrained RAM

Why Do This?

Before we move on, the important question we need to answer is WHY? Are there any practical applications for an LLM on our wrist? We like to think so:

• We demonstrate a Draw based Mood Uplifting Agent on Watch -> Free and Private
• Privacy preserving Chat Based User Interface for any wearable app as a simple, free SDK
  • Our framework can be drag-dropped into any supported XCode Project
  • We also cross compile to Linux aarch64 to allow for SDKs on Android or other distros
• Other Device based Applications like Privacy preserving Financial, Health, Messages Summarization and Tabulation are all unlocked through a Zero Trust framework usable in as low as 100 MB of peak RAM usage.

Our Solutions

• ARM NEON based SIMD optimizations for running the model with Performix benchmarking on hotpath
  • CLI that summarizes Performix code_hotspots and system_utilization reports.
  • Our CLI essentially identifies bottlenecks at the symbolic level in dequantization or matmul loops.
  • We basically tried a lot of things, but only report what "stuck" so to speak. Our graveyard is provided as an appendix.
• Fully mmapped model lazy loading layers into RAM only when needed
  • We used Zig's built in embedding and sequential execution for our token generation loop
  • Enabled the experimental @_extern flag to directly import our functions into Swift
  • All functions are called using unsafe pointers, with memory management in the Zig layer
• From scratch implementation for GGUF inference (we do not use any popular libraries)
  • Our inspiration is the legendary llama.cpp, but implementation is original and in Zig not C/C++
  • Our implementation is simplified and hackable compared to llama.cpp unlocking fast iteration
• From scratch implementation for CNN inference for touch drag emoji prompt generation
  • We code a simplified emoji identifier based on Conv2D (im2col) and MaxPool layers in Zig
  • We generate emoji shapes using Perfect Freehand and train our model in Pytorch
  • Fast way to prompt above LLM on watch, with a Keyboard fallback for actual typing
• Finetuned LLM to propose another application on ARM servers
  • We finetune Falcon H1 using SFT on Colab A100 (<$1 of compute needed) on PII masking data
  • We cross-compile our app to linux-aarch64 instead of watchos-aarch64

Optimization Path

1. Narrow Scoping and Choice of Tools

This was the most important part of our optimization journey imho. Since we were building for a hackathon, and are a small 2 member team, we needed as many "free gifts" as possible. Here's what we chose:

• Arm Performix for model evaluation gave us "free" symbolic benchmarking and insights
• Zig language gave us "free" fast cross compile and natively supported simd math
• SwiftUI gave us "free" drawable canvas and serialization for the touch interface
• Falcon H1 gave us "free" Mamba2 SSM O(n) attention vs SDPA O(n^2) attention

Our scope was purely to reduce the amount of memory required for inference while keeping Tokens per second output reasonable enough on an apple watch.

2. Choosing our Quantization appetite for constrained inference

We iterated across a bunch of quantizable models in the Falcon family, including Falcon H1 Tiny and Tiny Multilingual (90M params, explicitly meant for deployment on edge), Falcon-H1-0.6B-R (with CoT reasoning) and Falcon H1 0.5B. Below is our reasoning.

• We chose the Falcon Family for Mamba attention that has cheaper memory than SDPA/Flash based attention (linear vs quadratic)
• 90M is too good to be true, we tested Q8_0 and Q5_K_M quantizations. However, it is incoherent, and "wild" in multi-turn conversations, and is limited to "Toy" applications as a general purpose chat agent, with explicit fine tuning required and that too for single-turn applications
• 0.6B-R is a reasoning agent, but as noted in the TII blog itself this leads to more complexity for very little gain.
  
  • Since we did not have an existing streaming library, we could not manage tokens well enough to also support CoT reasoning within scope of our tiny model.
  • Also the blog supports performance gain equivalent to 7B models on reasoning, but we could not reproduce this on our general purpose chat testing. Maybe quantization affects the model adversely compared to the 0.5B non reasoning model.
• 0.5B does not support CoT reasoning, but shows strong multi turn performance, and we evaluated Q5_K_M before going with Q8_0 as our final model to optimize with our goals being stronger performance with as cheap peak memory as the Q5_K_M version

3. 8 Bit versus 5 bit Mixed Quantization versus 6bit Quantization SPEED/RAM

We used Llama.cpp to generate our Quantized models. This is industry standard and we did not find any need to try to optimize the quantization process.

• Convert Falcon H1 0.5 B (instruction tuned) to an F16 GGUF -> quantized version
• We iterated on the 5 bit and 6bit and 8 bit versions to determine the best used for our purposes
• Since we do not stream (no native support in WatchOS) we use TTAT (Time to All Tokens) not TTFT
• 6bit was quickly dropped as the worst of both worlds, low coherence plus larger footprint and worse speed

Here's our initial levels (post quantization and simd inference described in the next point):

| Metric                             | Q5_K (Nibble) | Q8_0 (Byte) |
|------------------------------------|--------------:|------------:|
| Model file size                    | 354 MB        | 531 MB      |
| Binary size                        | 357 MB        | 535 MB      |
| Gen time (generate_conversation)   | ~3271 ms      | ~2067 ms    |
| Memory Δ (RSS)                     | ~332 MB       | ~497 MB     |
| tok/s (approx)                     | ~3.1          | ~4.8        |

Arm Performix was helpful here since it provides us a symbolic level inference hotspot. We utilized only code_hotspots and system_utilization since we ran into tool_integrations.neoprof.INSUFFICIENT_PMU_COUNTERS upon trying the other recipes. Our CLI mapped the csv output to actual line numbers to edit in the given files.

KEY INSIGHT From Performix we derived this key insight: We can optimize the Hot Loop of Q8_0 quantized model to use almost the same amount of RAM as Q5_K_M while being nearly twice as fast by not having to inner loop dequantize on every matmul.

Below are the Q5_K_M quantized model running on our framework vs the Q8_0 model running on our framework. We use the following commands to run Performix:

apx recipe run code_hotspots --workload "echo 'What is the capital of India?' | ./llm-linux" --use-shell --deploy-tools

Followed by export apx run export <run-id> .

Before Loading to our cli for symbolic info ./pxy performix/reports/code_hotspots_3ac75e10feeb.zip

Q5_K_M quantization - Report Abridged for Easy Reading

PXY - Performix Profiling Report
═══════════════════════════
Workload:   echo 'What is the capital of India?' | ./llm-linux
Target:     localhost
Engine:     1.19.0
Recipe:     code_hotspots
Duration:   2026-07-25T10:20:30Z  →  2026-07-25T10:20:42Z
            12s total

Flat Function Ranking
────────────────────────────────────────────────────────────
 #  Samps      %   Cum%  Payoff  Function                                        
 1   4671   67.4   67.4  *****   dequant.deqRowNibble                            
 2    812   11.7   79.1  ***     mats.matmulQuantized                            
 3    626    9.0   88.2  **      mats.matmulQuantizedBatch                       
 4    387    5.6   93.7  **      genny.mamba2Layer                               
 5    248    3.6   97.3  *       dequant.deqRowWord 

Source Hotspots
────────────────────────────────────────────────────────────
  dequant.deqRowNibble  [ 67.4%, 8.1s]  P0
    dequant.zig:  91   14.4%  ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
    dequant.zig:  88   11.5%  ||||||||||||||||||||||||||||||||||||||||||||||||            
    dequant.zig: 100    7.6%  |||||||||||||||||||||||||||||||  

Q8_0 quantization - Report Abridged for Easy Reading

PXY - Performix Profiling Report
═══════════════════════════
Workload:   echo 'What is the capital of India?' | ./llm-linux
Target:     localhost
Engine:     1.19.0
Recipe:     code_hotspots
Duration:   2026-07-26T06:39:28Z  →  2026-07-26T06:39:39Z
            11s total

Flat Function Ranking
────────────────────────────────────────────────────────────
 #  Samps      %   Cum%  Payoff  Function                                        
 1   4795   82.4   82.4  *****   mats.matmulQuantized                            
 2    425    7.3   89.8  **      mats.matmulQuantizedBatch                       
 3    423    7.3   97.0  **      genny.mamba2Layer                               
 4     77    1.3   98.3  *       math.ldexp.ldexp__anon_6776                     
 5     27    0.5   98.8          compiler_rt.exp.exp      

Source Hotspots
────────────────────────────────────────────────────────────
  mats.matmulQuantized  [ 82.4%, 9.1s]  P0
    mats.zig: 487   82.4%  ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
    mats.zig: 194   22.9%  ||||||||||||||||                                            
    mats.zig: 192   14.5%  |||||||||| 

System Utilization (ASCT)

PXY - Performix Profiling Report
═══════════════════════════
Workload:   echo 'What is the capital of India?' | ./llm-linux
Target:     localhost
Engine:     1.19.0
Recipe:     system_utilization
Duration:   2026-07-26T07:02:51Z  →  2026-07-26T07:02:58Z
            7s total

System Utilization Timeline
────────────────────────────────────────────────────────────
  CPU:       79.8%  |||||||||||||||||||||||||||||||  (pinned)
  Memory:    15.1%  |||||||||||||||||||||||||||||||  (591MB / 3919MB)
  I/O wait:    0.0%  no blocking
  Swap:     none
  Procs:    1 running, 179 threads  (958 ctx/s)
  Disk I/O:  active  (read during startup)

Timeline (6 samples, 7s window)
  t=1s  CPU ################  Mem 14.8%  PF   9k  IO   0.0%
  t=2s  CPU ################  Mem 15.1%  PF   7k  IO   0.0%

Obviously we preferred 8bit quantization over 5bit or 6bit. The model is more coherent, follows instructions better (much better for smaller models <1B params) and responds more accurately to prompts in 8bit vs 5bit. Our blocker is now purely RAM (since watchOS, and pretty much any wearable OS) will kill the process if it consumes 100MB of peak memory or more.

3a. ARM scalar vs simd operations to improve inference speed with similar RAM usage

SIMD is a no brainer, and we followed a systematic approach towards implementing it. Our code is basically comprised of dequantization operations followed by multiple matmul operations. We wrote the code as scalar ops first, then used Opencode free AI agents in order to quickly convert to SIMD.

• Example: for (0..8) \|i\| out[i] = a[i] + b[i] is simply converted to const out: @Vector(8, f32) = a + b
  
• AI can generate Assembly instructions for this:
  • For scalar code: LDRB × 4 → AND × 4 → ... → SCVTF × 4 → FMUL × 4 → FSUB × 4 → STR × 4 (≈32 instr)
  • Versus for SIMD code: LD1.16B → AND.16B → CMTST.16B → ADD.16B → SCVTF.4S → FMUL.4S → FSUB.4S → ST1.4S (8 instr)
• We vectorize the code in Zig for dequantize first (this is trivial) and then compare generation time
• We then vectorize for matmuls (facing some issues with type casts) and then compare generation time

Overall, SIMD vs Scalar operations reduces TTAT by almost 50% with no memory changes. While benchmarking we get about the same RAM consumed but much faster inference. Since Q8_0 requires fewer dequant operations, and the matmuls are "simpler" it is also much faster, we get a further 50% reduction to TTAT for free by choosing Q8_0 instead of mixed bit quantization.

We also do SIMD type conversion for weights rather than process scalar, and use other free pragmas like @select and @shuffle and @mulAdd for further TTAT optimizations.

| Metric            | Q5_K (Nibble) | Q8_0 (Byte) | Speedup |
|-------------------|--------------:|------------:|---------:|
| Total time        | 2979 ms       | 1628 ms     | 1.83×    |
| Tokens processed  | ~10           | ~10         | —        |
| Tokens/sec        | ~3.4 tok/s    | ~6.1 tok/s  | 1.83×    |

3b. Demand paging usage exploiting mmap

We embed the entire 531 MB model directly into the binary via @embedFile, which means the full model is loaded into RAM at process start and never released. Since Q8_0 already pushes ~497 MB RSS delta, and even Q5_K sits at ~332 MB, both are well over the watchOS limit.

• Obvious alternative is mmap. In this case, we get a pseudo-mmap thanks to Swift. When Swift loads externC (behind experimental flag) it uses mmap to read our framework.
• First, the initial load drops from 535 MB to just the code — roughly 4 MB — making deployment practical for a wearable.
• Second, and more importantly, this pseudo mmap provides lazy page-in: the OS loads only the pages of the model that are actually touched during inference. For a short conversation where only a few layers' weights are accessed, a significant fraction of the model may never be faulted into RAM.
• Under memory pressure, watchOS can also evict clean mmap'd pages and re-read them from disk on demand, which is far more graceful than being killed.
• We also hook into memory pressure at the App level using DispatchSource.makeMemoryPressureSource but in practice this is never triggered
• Our final framework uses only about 64 MB peak RAM

| Metric              | Q8_0 (no mmap) | Q8_0 (mmap) |
|---------------------|---------------:|------------:|
| Init Load         | 535 MB         | ~4 MB       |
| Peak RSS            | ~497 MB        | ~64 MB      |
| Viable on watchOS   | ❌             | ✅          |

1. Packaging and Deployment We build a nominal toolchain on bash and zig in order to cross compile and deploy to linux-aarch64 , watchos-simulator-aarch64, watchos-aarch64 and macos-aarch64

We deploy to both simulator and watchOS since iterating on the actual device is cumbersome thanks to XCode issues.

1. Stuff that did not work (Graveyard)
2. Widening of SIMD vectorization (NEON is still limited by the max 128 bit size of registers)
3. Metal shaders programming to run on Watchos, since there is no GPU metal is filtered out by Swift
4. SmolLM2 family of models using standard Self Attention (SDPA) were incoherent for multi-turn
5. Precomputing dequantized weights at load — Q8_0 goes from 497 MB to ~1.5 GB
6. Comptime function dispatch for deqRow < 1 us difference at runtime
7. @Vector(16, f32) matmul inner loop - LLVM splits across 4 NEON registers and spills to stack
8. Q5_K-only quantization (no mixed Q6_K) lost accuracy on tensors — especially attn_v, ffn_down

Conclusion and Future Work

First off, thanks for reading this far. This is the most serious submission we have put into any hackathon ever, we did this not for fun, but to demonstrate an actual LLM on locally running apps on wearables. The submission for the hackathon itself is not rigorous enough to be a publication, however we do intend to try to scale this to a much much larger model and thoroughly benchmark for an actual publication later. We believe Arm Performix might be crucial for such a case. Until then, please check out our code and "tricks" for optimization. Thanks again!

• Team Pixy

Built With

• arm
• gguf
• llm