LFM2.5-24B-A2B MoE on Qualcomm IQ9075 NPU via KernelX
Liquid AI’s LFM2.5-24B-A2B stores 24 billion parameters but activates only about 2 billion for each token: 4 of 64 experts in each mixture-of-experts layer. That makes it a good candidate for edge hardware, as long as the runtime keeps the NPU busy. We ran the whole model on the Hexagon NPUs of a Qualcomm IQ-9075 board with llama.cpp. Upstream llama.cpp was the baseline, compared against two builds whose NPU kernels we developed with KernelX. We measured prefill and decode speed at six context lengths from 128 to 4,096 tokens, and runtime memory and perplexity at 512 tokens.
Test configurations
All three configurations load the same file, Liquid AI’s QAD-Q4_0 checkpoint (13.6 GiB). Q4_0 stores weights as 4-bit integers in blocks of 32 values, with one 16-bit scale per block. The label fp16 describes how the NPU computes with those weights. Each block is expanded to 16-bit floating point and multiplied on the NPU’s matrix engine. Our fp16 build is faster because its kernels are tuned for this NPU and it scores same perplexity as upstream. Our custom build incorporates fine tuned kernels using KernelX that make extensive use of the NPU’s matrix hardware and VTCM, while keeping perplexity at the fp16 level. All three get the same compute: both of the board’s NPUs, with the model’s 40 layers split across multiple sessions on each NPU. Every operation except the first token-embedding lookup runs on the NPU, and all runs used the same settings.
Prefill and decode speed
Figure 1. Prefill and decode speed
Each point is the mean of three llama-bench runs, with the model split across both NPUs.
At prefill our fp16 build runs at 1.27–1.30× the speed of upstream across the sweep with identical math. The custom build reaches 1.36–1.50×. For a 512-token prompt it processes 465 tokens per second against upstream’s 311, so the prompt is ready in 1.10 s instead of 1.65 s.
Decode speed sets how fast the answer streams out. With 128 tokens of context the custom build generates 22.6 tokens per second against upstream’s 16.6. Across all six context lengths the custom build decodes at 1.19–1.40× upstream’s speed, and our fp16 build at 1.10–1.26×.
Runtime memory and perplexity
Table 1. Runtime memory and perplexity at 512 tokens
| Configuration | Peak runtime memory | Perplexity | PPL vs upstream |
|---|---|---|---|
| ■ Upstream llama.cpp, fp16 | 14.0 GiB | 14.51 | – |
| ■ Ours, fp16 | 14.2 GiB | 14.50 | −0.1% |
| ■ Ours, custom | 15.0 GiB | 14.71 | +1.4% |
Peak runtime memory is the most system memory a run takes for a 512-token prompt and a 128-token answer, including the buffers shared with the NPU. Perplexity is computed over the entire WikiText-2 test set with a 512-token window.
Perplexity is maintained. Upstream scores 14.51, our fp16 build 14.50 and our custom build 14.71, within 1.4% of upstream. Runtime memory is mostly the 13.6 GiB of weights: 14.0 GiB for upstream, 14.2 GiB for our fp16 build and 15.0 GiB for the custom build, which needs 0.9 GiB more than upstream and stays well inside the board’s 34.5 GiB.
Test setup
Qualcomm IQ-9075 (QCS9075, 8 Arm cores, two Hexagon NPUs, 34.5 GiB RAM). Model LFM2.5-24B-A2B-QAD-Q4_0 from Liquid AI. Upstream llama.cpp at commit 23deec025 with its default NPU backend. Our builds are based on llama.cpp with NPU kernels developed using KernelX.