It's very difficult to implement efficient small-batch kernels. For speculative decoding, your best bet is to to increase the min draft size and keep the draft prob high:

--draft-max 16 --draft-min 8 --draft-p-min 0.9

This should still give you some speed-up for low-entropy generations.

Given the fact that even at batchsize 8 it's performing worse at token generation, that 'draft-min' should be even higher than 8, right? Especially given that it's unlikely all tokens in a long draft sequence will be accepted.

I understand that it's very difficult to optimize small batch kernels, and performance can actually go down compared to the non-batched case due to overhead, but an almost 4x drop in performance going from non-batched to batchsize 2 sounds like a bug / major bottleneck somewhere, right?

Just confirmed my own thoughts. Using:

llama-speculative-simple-vulkan -m /models/Qwen2.5-Coder-32B-Instruct-Q4_K_S.gguf -md /models/Qwen2.5-Coder-0.5B-Instruct-IQ4_XS.gguf -ngl 99 -ngld 99 -p "Write a minesweeper game using html, js and css. Do not give any explanations. Only output the code." --draft-p-min 0.9 --draft 16 --draft-min 8

I am seeing just over 30 tokens/sec:

decoded 1325 tokens in 43.903 seconds, speed: 30.180 t/s

The quality of the draft looked good as expected from the settings used:

n_draft   = 16
n_predict = 1325
n_drafted = 655
n_accept  = 631
accept    = 96.336%

But a simply llama-cli used as follows:

llama-cli-vulkan -m /models/Qwen2.5-Coder-32B-Instruct-Q4_K_S.gguf -ngl 99 -p "Write a minesweeper game using html, js and css. Do not give any explanations. Only output the code."

Is going at above 33 tokens/sec:

llama_perf_context_print: eval time = 28566.51 ms / 956 runs ( 29.88 ms per token, 33.47 tokens per second)

The Vulkan backend has two paths for matrix multiplication - a matrix-vector multiply for when N=1, and a matrix-matrix multiply for N>1. The matrix-matrix multiply is really aimed at larger matrices, and doesn't do well with small N. We should be able to do better by adapting the matrix-vector multiply to be able to do a few vectors at a time. I can look into this soon, but we should probably let #10846 land first to avoid conflicts.

Awesome, thanks for your explanation! Let me know when you start working on this, I'd be happy to help benchmark certain setups to see what works best. I know it's only applicable to the 7900xtx, but maybe it's useful data for you.

Hi @Mushoz, please give #10991 a try when you get a chance.

@jeffbolznv Will give feedback in that PR to keep the discussion in one place