Making LFM2.5-2.6B fast on a Ryzen AI 9 HX 370Local inference benchmark
Making LFM2.5-2.6B fast
The same 2.6B model, same laptop, measured across three quantizations and two
backends. The starting point was PyTorch transformers at bfloat16; everything
after that is llama.cpp b10280.
Fastest generation
43.4 tok/s
Speedup vs start
6.2×
Fastest prefill
1,106 tok/s
Token generation
What you feel while the model is typing. Quantization dominates: every halving
of weight size buys back nearly proportional speed, because each token must read
every weight from memory.
How fast a long prompt is ingested before the first token appears. This stage is
compute-bound rather than memory-bound, which is why the integrated GPU pulls
far ahead here and barely ahead above.
llama.cpp CPUllama.cpp iGPU (Vulkan)
Q4_K_M · 1.56 GiB
CPU
Q4_K_M on CPU — 233.2 tok/s · 1.56 GiB of weights
233 tok/s
iGPU
Q4_K_M on iGPU — 1,095.3 tok/s · 1.56 GiB of weights
1,095 tok/s
Q8_0 · 2.68 GiB
CPU
Q8_0 on CPU — 195.7 tok/s · 2.68 GiB of weights
196 tok/s
iGPU
Q8_0 on iGPU — 1,106.3 tok/s · 2.68 GiB of weights
1,106 tok/s
BF16 · 5.03 GiB
CPU
BF16 on CPU — 167.2 tok/s · 5.03 GiB of weights
167 tok/s
iGPU
BF16 on iGPU — 467.2 tok/s · 5.03 GiB of weights
467 tok/s
05641,128tok/s
Why quantization is the whole story
Multiplying weight size by generation rate gives the memory bandwidth each
configuration actually achieves. Every llama.cpp build lands on the same ceiling
regardless of quantization — they are all reading memory as fast as this machine
physically can, so the only remaining lever is reading less of it.
measured ceiling 70 GB/s
Q4_K_M CPU
Q4_K_M on CPU — reading 1.56 GiB × 38.5 tok/s
60 GB/s
Q4_K_M iGPU
Q4_K_M on iGPU — reading 1.56 GiB × 43.4 tok/s
68 GB/s
Q8_0 CPU
Q8_0 on CPU — reading 2.68 GiB × 23.0 tok/s
62 GB/s
Q8_0 iGPU
Q8_0 on iGPU — reading 2.68 GiB × 26.0 tok/s
70 GB/s
BF16 CPU
BF16 on CPU — reading 5.03 GiB × 12.8 tok/s
64 GB/s
BF16 iGPU
BF16 on iGPU — reading 5.03 GiB × 14.3 tok/s
72 GB/s
04079GB/s
Thread count
More threads stop helping once memory is saturated. Past
8 threads the cores simply queue for the same bandwidth,
and using all 24 SMT threads is slower than using four.
4 thr
4 threads — 27.8 tok/s
27.8 tok/s
8 thr
8 threads — 39.0 tok/s (fastest)
39.0 tok/s
12 thr
12 threads — 38.3 tok/s
38.3 tok/s
16 thr
16 threads — 37.5 tok/s
37.5 tok/s
24 thr
24 threads — 25.3 tok/s
25.3 tok/s
02040tok/s
Every measurement
Build
Backend
Weights (GiB)
Prefill (tok/s)
Generate (tok/s)
Eff. BW (GB/s)
Q4_K_M
CPU
1.56
233
38.5
60
Q4_K_M
iGPU
1.56
1,095
43.4
68
Q8_0
CPU
2.68
196
23.0
62
Q8_0
iGPU
2.68
1,106
26.0
70
BF16
CPU
5.03
167
12.8
64
BF16
iGPU
5.03
467
14.3
72
transformers bfloat16
CPU
5.03
—
7.0
35
What this means
Use Q4_K_M. It is 6.2×
faster than where we started and small enough to leave most of your RAM free.
Parameter count is a red herring. A 2.6B model at bfloat16 reads 5.03 GiB
per token — more than an 8B model at 4-bit. Bytes per token is what sets speed,
not billions of parameters.
The runtime mattered as much as the format. At identical bfloat16
weights, llama.cpp reached
12.8
tok/s against transformers' 7.0 — same math, better kernels.
The iGPU is for prompts, not for typing. Vulkan gives a
5.7×
prefill win but only a few tok/s on generation, because it shares the same memory.