Story 007 - Making LLM Inference Faster Led Me Down the Performance Stack

“Make the agent faster.” That was essentially the requirement.

The agent used an open-weight reasoning model and was deployed on an AWS EC2 instance with a single NVIDIA A10G. It was the chatbot I built in Story 000, which had since evolved into an agent (Story 004). The majority of the user-visible latency came from the multi-step LLM generations required to fulfill a single user request.

User Request ---> Planner LLM Reasons & Generates Tool Calls ---> Orchestrator executes tool calls ---> Enterprise Application API calls ---> Generator LLM receives tool results, reasons, and then streams the first user-visible token

Of course I had streaming tokens enabled, but it was still not enough for the managers. So I changed the UX to show the reasoning tokens in a mini box, similarly to how LM Studio’s GUI does it. The idea was that the user could see beyond the opaque “Thinking…” text and at least know that something was happening. The managers didn’t like it. The user base was not even slightly tech-savvy, and they felt the reasoning display would make the experience more confusing. So I reverted the change.

Now what?

If the model was going to generate a long reasoning trace, I couldn’t simply make the text appear faster by wishing it into existence. The network-bound external API calls did add some noticeable latency, but they were negligible compared to the Planner and Final LLM reasoning completion. The autoregressive generation rate was around ~90 tok/s.

Thank god I had this performance optimization problem, because it turned me into something more than a mere vLLM inference-engine user and sent me down the currently infant inference-engineering rabbit hole. I did my research and discovered speculative decoding, which promised roughly 2× faster generation under favorable conditions. And, thanks to the coincidence that I was using Gemma 4 E4B IT, there was a readily available dedicated MTP drafter model trained by Google alongside the model distribution I was already using. I had actually stumbled upon that assistant checkpoint earlier, when I was looking for models to upgrade the chatbot into this assistant, but its purpose had not been clear to me then.

After clearing speculative decoding at the theoretical concept level and gathering everything I needed for a smoke test, I was ready to try it on my Workstation. I passed the newly learned speculative config:mtp ... vLLM args, and the new MTP vLLM deployment was live. I sent a couple of requests and saw in the vLLM telemetry that the average generation rate had bumped up to ~200 tok/s.

Damn. I felt like I had strapped a supercharger onto my car.

But then I had the question: Why?

I had made inference faster. Cool. But what had actually changed underneath?

The vLLM telemetry could tell me the increase in generation rate. It couldn’t tell me what the GPU was doing differently. So I started digging. My first stop was nvtop. The GPU was busy. Great. But that didn’t explain the speedup. GPU utilization is a useful signal, but it is still just a signal. I needed to know what that utilization actually consisted of.

So I learnt and started GPU profiling, then opened NVIDIA Nsight Systems. And this was where things got serious. I was suddenly looking at CPU activity, CUDA launches, CUDA Graphs, GPU execution timelines, kernel invocations, gaps between executions, and an absurd amount of information that I had never needed to care about when I was simply using vLLM as an inference engine.

I zoomed in.
Then I zoomed in again.
It was an infinite zoom, like entering the Quantum Realm of Ant-Man.
In that process, I realized that I was no longer looking at “GPU utilization.” I was looking at the execution of the inference engine itself.

But I still didn’t understand the whole picture. So I went one layer deeper into the runtime and started using PyTorch Profiler. Now I could see the runtime path around the GPU work: model execution, sampling, the speculative proposal path, and the additional work that MTP was introducing. The MTP execution was clearly more complicated. It wasn’t simply the same inference loop running faster.

In fact, the profiler and Nsight were showing me the opposite: each MTP execution carried more work.

Okay.

Then how the hell was it faster? That question sent me one layer deeper.

Nsight Compute. This was where I was properly out of my depth.

Now I was staring at GEMMs and GEMVs, memory throughput, SM activity, occupancy, active warps, registers per thread, shared memory, cache hit rates, instruction throughput…

I had gone from looking at a chatbot’s response time to looking at individual GPU kernels and hardware counters.

And honestly, I was overwhelmed. But I had already come this far.

So I started learning what the numbers actually meant, one piece at a time, and connecting the evidence from each layer instead of trying to understand everything at once.

The first important clue came from the Nsight Systems traces. MTP was doing something more complicated than autoregressive decoding, but it was also repeating the expensive execution pattern substantially fewer times per generated token. Across the workloads, the selected repeating CUDA Graph executions fell by 56.4–78.1% per generated token.

Okay. That was interesting. Maybe the answer wasn’t that MTP made each execution cheaper. Maybe it was making each execution count for more tokens.

The PyTorch Profiler traces gave me the other side of the picture. MTP was indeed paying extra cost: proposal generation, sampling, candidate processing, and additional runtime and GPU work. The median GPU-context span was 58.5–61.5% longer under MTP, while CPU-context spans were also longer.

So MTP wasn’t some magically faster version of the same execution. It was a more expensive execution that produced more useful token progress.

Then I went all the way down to the kernels. The dominant autoregressive kernel was a GEMV. The dominant MTP kernel was a GEMM.

Surely the GEMM was faster, right?

No.

The representative MTP GEMM took about 2.79 ms. The representative AR GEMV took about 2.78 ms. Almost identical. And both were operating near the A10G’s 600 GB/s memory-bandwidth ceiling: roughly 575 GB/s for the selected MTP GEMM and 579 GB/s for the selected AR GEMV.

So the answer wasn’t hiding in some faster kernel. The MTP GEMM wasn’t faster.

The GPU wasn’t suddenly becoming more efficient at executing the individual operation.

MTP was winning somewhere else.

It was doing more useful token progress for each expensive recurring execution. And that was the moment the pieces finally started connecting.

The acceptance telemetry told me that speculative proposals were producing multiple tokens of useful progress per verification iteration. Nsight Systems showed that the recurring GPU execution happened fewer times per generated token. PyTorch Profiler showed me the additional speculative work that MTP had to pay for. And Nsight Compute showed me that the dominant kernel itself wasn’t faster.

The speedup was coming from amortization.

The additional speculative work was real. The individual MTP execution could even be more expensive. But that cost was being spread across more useful generated tokens, reducing how often the expensive recurring GPU work had to be performed.

That was the answer I had been looking for. The faster path was not a faster kernel.

And somewhere along the way, I had stopped trying to make inference faster. I was trying to understand it.

That investigation became the technical report Beneath the Tokens: A Performance Engineering Study of Multi-Token Prediction in GPU-Accelerated LLM Inference.