Writings

I read a lot of articles complaining about AI speed benchmarks. The common refrain is that these benchmarks exist and are so commonly accepted that the only way to get traction with your model is for it to do well on those tests. So creators train their models to do well on those tests, in a process now known as "benchmaxxing". In theory there's nothing wrong with this - if there's a test, let's make sure we can do well on it. But in reality there's a real issue with this practice: spending time training a model to pass a benchmark means you're drifting the model away from what it should be good at instead. So the complaint is a valid one - the benchmarks are actually making models worse instead of better. But humans need to understand everything as a comparison, so the benchmarks are a necessary evil. There isn't really a good answer there.

Despite all this, that isn't even my problem. I have different issues to talk about. When models are benchmarked for speed, they are generally using the measurement of "tokens per second" (tok/s). The main problem I have is that almost all benchmarks on AI speed are on Token Generation (TG). When you see headlines like: "This model gets 50 tok/s!" unless qualified, these will always refer to TG tok/s. And TG is a big deal - this is the speed that a model outputs content back to the user. So it's not only a useful metric, but an important one.

My problem isn't that this exists, or that it is highlighted. I have two issues with this, and much like everything else on AI, my problems are with the practical implications:

Issue #1: TG benchmarks are also benchmaxxed.

You can tune a model to output at a higher speed than what it would do in any real-world scenario. If you turn the temperature of a model down to .2 or 0, it gets far less creative in its thinking, so it will spend less effort on its answers. And because it's generating far fewer tokens, it spits them out at a much higher rate. You can further increase this by limiting context size, limiting max output tokens, and increasing speculative decoding to a higher rate (5-7 or even higher in rare cases) to eke out every last token per second.

The problem I have is this isn't indicative of real world performance. When you turn the sampling and context settings back to a real world use case (temp in the .7 to 1 range, context size to over 128k, etc.) the tok/s for TG drops significantly. Similarly you can get much higher speeds out of a more heavily quantized model - a Q4 quant of a model will be inherently faster than a Q6, even though it will be less coherent. If the model needs four extra turns to get a script correct at Q4, it's irrelevant how fast it is. And this is fine if you know these things well, but the problem lies in the fact that people new to AI now have a skewed perception of the kind of responsiveness and speed they should expect from the models they are using.

When I first started setting up local AI inference, I read all the benchmarks and articles and I expected to get ~35 tok/s from my model. Then I ran it with all the recommended settings and was getting 8 tok/s. If you're learning AI for the first time, this is a real shock to the system.

I do understand that it's very difficult to get a practical measurement of TG tok/s because use cases will greatly affect the outcomes here. Chatting with an agent will get much lower TG than coding. Solving a problem will get lower TG than creating something new. Every use case is different, so yes this is subjective. But still - you can give more realistic expectations if you have show measurements for different use cases. Keep the "this is the max speed" baseline as we have today. But add a coding speed and a chat speed that can give people a better expectation of how responsive the model will be for their tasks.

Issue #2: TG is the wrong measurement in most cases.

Again, TG is important. But agentic is now the most common use case for AI. And something that's not often talked about in this context is the fact that most common AI agent harnesses pass a lot of tokens into the model in every turn. It's not uncommon for a typical interaction with an AI agent to have over 100,000 tokens sent in and under 1,000 returned. So the model has to process significantly more input tokens than output. This is known as Prompt Processing (PP) and has a completely different throughput than TG.

There are two misleading factors with not measuring PP as part of the baseline "speed" of a model. First is that the settings I referenced that increase TG can negatively impact PP. Speculative decoding materially increases TG but reduces PP throughput. And second, lowering context size to 8192 for a TG benchmark is irrelevant when most agents barely function with a context size less than 128k. (Yes, you can have context significantly smaller for coding-only, but actual agentic work requires a much higher window.)

The reality of this is that "wall time" (the amount of time from beginning to end of a turn) is made up much more of PP than TG in most agentic use cases. If you send in a prompt to an agent and it takes 20 seconds to respond, many cases would have that be 15 seconds for PP and 5 for TG. Mind you - PP and TG have individual dials you can tune (as well as cache reuse which can heavily impact wall time). So while wall time is the perceived response time, there are a lot of factors that go into it, and some that have zero to do with the model itself. So PP is still the more useful benchmark for agentic use.

I will concede two issues with my push for PP as a more impactful speed measurement:

First, inference providers can have a big impact on PP. Two of the most popular ones - llama.cpp and vLLM - each deal with throughput differently. Llama.cpp is often regarded as the "faster" provider because you can get higher TG speeds out of it in many cases. But vLLM has a significantly higher PP in most cases, most especially if you have parallel calls coming in. So people new to local AI will see the benchmarks that llama.cpp is "faster" and then see their wall time being much slower than it should be.

Second, some newer models are made to do heavy reasoning to get to the right answer in fewer turns. There's a stigma in the community around these models, calling them "overthinkers". I won't comment on whether this is good or bad, but it is an intentional shift in how some of these models behave. The newly released Qwen 3.8 27b from Alibaba can easily crack over 10,000 reasoning tokens before giving an answer. And Poolside's Laguna S 2.1 routinely burns over 40,000 reasoning tokens per turn. In these cases, yes TG is an impactful measurement of their responsiveness.

So where does this leave us? The solution is pretty simple: as with almost everything else - more information is better than less. The more input we have into any decision, the better decisions we can make. In an ideal world you'd see something like this for speed benchmarks:

Provider 	64k PP 		Chat TG 	Coding TG

Llama.cpp	300t/s		30t/s		70t/s

vLLM		1,200t/s	25t/s		60t/s

You might still have the issue of benchmaxxing, but it's a start. Having a rotating list of prompts to use for these could solve that as you can then use averages across them. It might not be fully apples-to-apples if you had to keep these revolving so model creators couldn't train for specific prompts, but I'd rather have slight drift in these numbers than know the models are going to be worse for it.

It's not a perfect solution by any means, but at least then we'd have a clearer picture of factors and tradeoffs to decide which model and which inference provider are right for our individual use cases.