Whenever you optimize the inference efficiency of an LLM, it’s essential know tips on how to measure it. With out measurement, it’s straightforward to make a mannequin extra difficult with out making it sooner, or to enhance throughput whereas making user-visible latency worse.
An LLM service has a number of sorts of efficiency. A consumer cares about how lengthy it takes to see the primary token and the way shortly the remainder of the reply streams. An operator cares about what number of requests the {hardware} can serve, how a lot reminiscence is used, and the way a lot every generated token prices. A researcher might care about whether or not an optimization modifications the mannequin’s output high quality.
On this chapter, you’ll find out about:
Latency and throughput metrics
Time to first token and time per output token
Measuring CPU and GPU inference
Utilizing CUDA occasions
Benchmarking a number of requests
Enthusiastic about a number of GPUs and a number of machines
Let’s get began.
Measuring Efficiency of Transformer InferencePhoto by Tomas Anton Escobar. Some rights reserved.
Overview
This chapter is split into eight components; they’re:
Metrics for LLM Inference
Measuring a Single Request
Warmup and Synchronization
Measuring GPU Work with CUDA Occasions
Measuring Reminiscence Utilization
Measuring Concurrent Requests
A number of GPUs and A number of Machines
Value per Token
Metrics for LLM Inference
The most typical inference metrics are:
Latency: How lengthy a request takes from begin to end.
Time to first token (TTFT): How lengthy the consumer waits earlier than the primary output token seems.
Time per output token (TPOT): The common time between generated tokens after the primary token.
Throughput: What number of tokens or requests are processed per second.
Reminiscence utilization: How a lot CPU reminiscence or GPU reminiscence is used.
Utilization: How busy the accelerator is in the course of the benchmark.
Value per token: The {hardware} or service price divided by the variety of tokens processed.
For LLMs, a single latency quantity is normally not sufficient. Take into account two requests:
Request A: 2,000 immediate tokens and 20 output tokens
Request B: 20 immediate tokens and a couple of,000 output tokens
Request A stresses prefill. Request B stresses decode. They could have the identical complete variety of tokens, however they’ve completely different efficiency profiles. That is why you must document immediate tokens and output tokens individually.
Tail latency additionally issues. If most requests full in a single second however a couple of take ten seconds, customers will discover. Report high-percentile latencies equivalent to p90, p95, and p99 along with the imply or median. The excessive percentiles describe the worst circumstances higher. You possibly can simply discover these percentiles from an inventory of values utilizing NumPy:
import numpy as np
def summarize(values):
values = np.asarray(values, dtype=np.float64)
return {
“imply”: values.imply(),
“median”: np.percentile(values, 50),
“p90”: np.percentile(values, 90),
“p95”: np.percentile(values, 95),
“p99”: np.percentile(values, 99),
}
import numpy as np
def summarize(values):
values = np.asarray(values, dtype=np.float64)
return {
“imply”: values.imply(),
“median”: np.percentile(values, 50),
“p90”: np.percentile(values, 90),
“p95”: np.percentile(values, 95),
“p99”: np.percentile(values, 99),
}
These numbers are easy, however they forestall a standard mistake: optimizing the typical whereas making the worst circumstances slower.
Measuring a Single Request
The best measurement makes use of time.perf_counter(). It’s a built-in high-resolution wall-clock timer appropriate for measuring elapsed time in Python. It’s extra correct than time.time().
The next instance measures prefill and decode individually for a Hugging Face causal language mannequin:
import time
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
def load_model(model_name=”sshleifer/tiny-gpt2″, machine=”cpu”):
tokenizer = AutoTokenizer.from_pretrained(model_name)
mannequin = AutoModelForCausalLM.from_pretrained(model_name).to(machine)
mannequin.eval()
return tokenizer, mannequin
@torch.no_grad()
def measure_one_request(mannequin, tokenizer, immediate, max_new_tokens=50, machine=”cpu”):
input_ids = tokenizer(immediate, return_tensors=”pt”).input_ids.to(machine)
begin = time.perf_counter()
outputs = mannequin(input_ids, use_cache=True)
prefill_end = time.perf_counter()
past_key_values = outputs.past_key_values
next_token = outputs.logits[:, -1, :].argmax(dim=-1, keepdim=True)
generated = [next_token]
decode_times = []
for _ in vary(max_new_tokens – 1):
step_start = time.perf_counter()
outputs = mannequin(
next_token,
past_key_values=past_key_values,
use_cache=True,
)
# Notice: Chances are you’ll want torch.cuda.synchronize() right here
step_end = time.perf_counter()
decode_times.append(step_end – step_start)
past_key_values = outputs.past_key_values
next_token = outputs.logits[:, -1, :].argmax(dim=-1, keepdim=True)
generated.append(next_token)
if tokenizer.eos_token_id just isn’t None:
if next_token.merchandise() == tokenizer.eos_token_id:
break
finish = time.perf_counter()
output_ids = torch.cat([input_ids] + generated, dim=1)
return {
“textual content”: tokenizer.decode(output_ids[0], skip_special_tokens=True),
“prompt_tokens”: input_ids.dimension(1),
“output_tokens”: len(generated),
“prefill_seconds”: prefill_end – begin,
“decode_seconds”: sum(decode_times),
“total_seconds”: finish – begin,
“ttft_seconds”: prefill_end – begin,
“seconds_per_output_token”: (
sum(decode_times) / max(1, len(decode_times))
),
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
import time
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
def load_model(model_name=“sshleifer/tiny-gpt2”, machine=“cpu”):
tokenizer = AutoTokenizer.from_pretrained(model_name)
mannequin = AutoModelForCausalLM.from_pretrained(model_name).to(machine)
mannequin.eval()
return tokenizer, mannequin
@torch.no_grad()
def measure_one_request(mannequin, tokenizer, immediate, max_new_tokens=50, machine=“cpu”):
input_ids = tokenizer(immediate, return_tensors=“pt”).input_ids.to(machine)
begin = time.perf_counter()
outputs = mannequin(input_ids, use_cache=True)
prefill_end = time.perf_counter()
past_key_values = outputs.past_key_values
next_token = outputs.logits[:, –1, :].argmax(dim=–1, keepdim=True)
generated = [next_token]
decode_times = []
for _ in vary(max_new_tokens – 1):
step_start = time.perf_counter()
outputs = mannequin(
next_token,
past_key_values=past_key_values,
use_cache=True,
)
# Notice: Chances are you’ll want torch.cuda.synchronize() right here
step_end = time.perf_counter()
decode_times.append(step_end – step_start)
past_key_values = outputs.past_key_values
next_token = outputs.logits[:, –1, :].argmax(dim=–1, keepdim=True)
generated.append(next_token)
if tokenizer.eos_token_id is not None:
if next_token.merchandise() == tokenizer.eos_token_id:
break
finish = time.perf_counter()
output_ids = torch.cat([input_ids] + generated, dim=1)
return {
“textual content”: tokenizer.decode(output_ids[0], skip_special_tokens=True),
“prompt_tokens”: input_ids.dimension(1),
“output_tokens”: len(generated),
“prefill_seconds”: prefill_end – begin,
“decode_seconds”: sum(decode_times),
“total_seconds”: finish – begin,
“ttft_seconds”: prefill_end – begin,
“seconds_per_output_token”: (
sum(decode_times) / max(1, len(decode_times))
),
}
This operate doesn’t use the mannequin’s generate() technique. That’s intentional. The purpose is to reveal prefill and decode to allow them to be measured individually. The variety of output_tokens contains the tokens generated by each prefill and decode. The seconds_per_output_token is the typical time per output token within the decode part.
There are two particulars to note:
use_cache=True asks the mannequin to return the KV cache.
Throughout decode, the mannequin receives solely next_token, not the entire sequence.
This is similar concept as Chapter 1, however utilizing a library mannequin.
Warmup and Synchronization
Whenever you measure efficiency, be aware that some one-time prices mustn’t dominate the outcome. In Python, import of a module might be sluggish however subsequent import of the identical module is instantaneous. Equally, the primary execution of some code could also be slower than subsequent executions as a result of initialization of information constructions or warmup of caches. You need to measure steady-state work, not that setup overhead.
Subsequently, benchmarks ought to embody warmup. The primary few iterations could also be slower for varied causes. As an alternative of measuring the overall time and dividing by the variety of iterations, you must measure the time for every iteration and analyze the steady-state ones. For instance, in the event you use the mannequin to generate a number of tokens, you’ll probably put the era in a loop. Measure every iteration as follows, then ignore the primary few outcomes:
def iterations(mannequin, tokenizer, immediate, machine, steps=100, warmup=10):
outcomes = []
for _ in vary(steps):
outcome = measure_one_request(
mannequin,
tokenizer,
immediate,
max_new_tokens=8,
machine=machine,
)
outcomes.append(outcome)
regular = outcomes[warmup:]
return summarize([item[“total_seconds”] for merchandise in regular])
def iterations(mannequin, tokenizer, immediate, machine, steps=100, warmup=10):
outcomes = []
for _ in vary(steps):
outcome = measure_one_request(
mannequin,
tokenizer,
immediate,
max_new_tokens=8,
machine=machine,
)
outcomes.append(outcome)
regular = outcomes[warmup:]
return summarize([item[“total_seconds”] for merchandise in regular])
Should you use GPU to run your LLM inference, you additionally must initialize the kernels while you first run them. Sadly, many GPU operations are asynchronous. That’s, when you launched an operation on GPU, Python might proceed along with your code instantly whereas the GPU remains to be working. Subsequently, a naive method to measure the time can be incorrect. As an alternative, you must use torch.cuda.synchronize() to attend for the GPU to complete the operation earlier than you cease the timer:
def sync_if_needed(machine):
if machine.startswith(“cuda”):
torch.cuda.synchronize()
begin = time.perf_counter()
outputs = mannequin(input_ids, use_cache=True)
sync_if_needed(machine)
elapsed = time.perf_counter() – begin
def sync_if_needed(machine):
if machine.startswith(“cuda”):
torch.cuda.synchronize()
begin = time.perf_counter()
outputs = mannequin(input_ids, use_cache=True)
sync_if_needed(machine)
elapsed = time.perf_counter() – begin
This provides a wall-clock measurement that features the precise GPU work. For correct prefill and per-token decode timings on GPU, name sync_if_needed(machine) after every timed mannequin(…) name in measure_one_request(), not solely as soon as on the finish of the request.
Measuring GPU Work with CUDA Occasions
CUDA occasions measure elapsed time the GPU spent executing kernels, not the end-to-end consumer latency. This time doesn’t embody any Python overhead. Under is an instance of tips on how to use CUDA occasions to measure the time:
def cuda_event_time(fn):
begin = torch.cuda.Occasion(enable_timing=True)
finish = torch.cuda.Occasion(enable_timing=True)
begin.document()
outcome = fn()
finish.document()
torch.cuda.synchronize()
milliseconds = begin.elapsed_time(finish)
return outcome, milliseconds / 1000.0
def cuda_event_time(fn):
begin = torch.cuda.Occasion(enable_timing=True)
finish = torch.cuda.Occasion(enable_timing=True)
begin.document()
outcome = fn()
finish.document()
torch.cuda.synchronize()
milliseconds = begin.elapsed_time(finish)
return outcome, milliseconds / 1000.0
You should utilize it to measure one ahead move:
with torch.no_grad():
outputs, seconds = cuda_event_time(
lambda: mannequin(input_ids, use_cache=True)
)
print(f”GPU ahead time: {seconds:.6f} seconds”)
with torch.no_grad():
outputs, seconds = cuda_event_time(
lambda: mannequin(input_ids, use_cache=True)
)
print(f“GPU ahead time: {seconds:.6f} seconds”)
CUDA occasion timing and wall-clock timing reply completely different questions:
Wall-clock timing measures what the appliance experiences.
CUDA occasion timing measures how lengthy the GPU work took.
For an inference service, wall-clock timing is normally the first metric as a result of customers expertise queues, tokenization, scheduling, community overhead, and streaming. CUDA occasions are helpful when you’re optimizing kernels or evaluating mannequin execution paths.
For deeper GPU profiling, use instruments equivalent to PyTorch Profiler, Nsight Methods, Nsight Compute, or CUPTI-based monitoring. These instruments can report kernel timelines, reminiscence copies, GPU utilization, and operator-level breakdowns. They’re extra complicated than a timer, however they’re crucial when a easy benchmark says the mannequin is sluggish and it’s essential know why.
Measuring Reminiscence Utilization
Reminiscence is a unique dimension to measure as a result of it limits not velocity for one consumer a lot as what number of customers your system can serve. Normally the GPU reminiscence is the bottleneck. In PyTorch, you’ll be able to report allotted and reserved reminiscence like the next:
def gpu_memory_summary(machine=”cuda”):
torch.cuda.synchronize()
return {
“allocated_gb”: torch.cuda.memory_allocated(machine) / 1e9,
“reserved_gb”: torch.cuda.memory_reserved(machine) / 1e9,
“max_allocated_gb”: torch.cuda.max_memory_allocated(machine) / 1e9,
}
def gpu_memory_summary(machine=“cuda”):
torch.cuda.synchronize()
return {
“allocated_gb”: torch.cuda.memory_allocated(machine) / 1e9,
“reserved_gb”: torch.cuda.memory_reserved(machine) / 1e9,
“max_allocated_gb”: torch.cuda.max_memory_allocated(machine) / 1e9,
}
The allotted worth is reminiscence utilized by tensors. The reserved worth is reminiscence held by PyTorch’s caching allocator. The utmost allotted worth is usually essentially the most helpful quantity for capability planning.
The allotted and reserved reminiscence are real-time snapshots however the max allotted worth is a peak over time. For correct measurement, you must reset the height statistic earlier than a benchmark:
torch.cuda.reset_peak_memory_stats()
outcome = measure_one_request(mannequin, tokenizer, immediate, machine=”cuda”)
reminiscence = gpu_memory_summary(“cuda”)
print(reminiscence)
torch.cuda.reset_peak_memory_stats()
outcome = measure_one_request(mannequin, tokenizer, immediate, machine=“cuda”)
reminiscence = gpu_memory_summary(“cuda”)
print(reminiscence)
Reminiscence ought to be measured along with tokens. A run with an extended immediate or extra generated tokens will naturally use extra KV cache reminiscence.
Measuring Concurrent Requests
To create a server that runs a language mannequin, consider the system by what number of requests you’ll be able to serve per second. Throughput will depend on each how briskly you fulfill one request and what number of requests you’ll be able to run concurrently, although concurrency doesn’t scale linearly beneath competition.
Manufacturing methods ought to deal with a number of customers, and the scheduler might batch their work collectively. The next easy benchmark runs a number of requests concurrently utilizing Python threads. This doesn’t implement steady batching. It solely measures how a mannequin wrapper behaves when a number of callers use it on the similar time.
from concurrent.futures import ThreadPoolExecutor, as_completed
def run_prompt(mannequin, tokenizer, immediate, machine):
begin = time.perf_counter()
outcome = measure_one_request(
mannequin,
tokenizer,
immediate,
max_new_tokens=32,
machine=machine,
)
finish = time.perf_counter()
outcome[“wall_seconds”] = finish – begin
return outcome
def benchmark_concurrent(mannequin, tokenizer, prompts, machine=”cpu”, employees=4):
outcomes = []
begin = time.perf_counter()
with ThreadPoolExecutor(max_workers=employees) as pool:
futures = [
pool.submit(run_prompt, model, tokenizer, prompt, device)
for prompt in prompts
]
for future in as_completed(futures):
outcomes.append(future.outcome())
finish = time.perf_counter()
total_output_tokens = sum(merchandise[“output_tokens”] for merchandise in outcomes)
return {
“requests”: len(outcomes),
“total_seconds”: finish – begin,
“output_tokens”: total_output_tokens,
“output_tokens_per_second”: total_output_tokens / (finish – begin),
“latency_summary”: summarize([item[“wall_seconds”] for merchandise in outcomes]),
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
from concurrent.futures import ThreadPoolExecutor, as_completed
def run_prompt(mannequin, tokenizer, immediate, machine):
begin = time.perf_counter()
outcome = measure_one_request(
mannequin,
tokenizer,
immediate,
max_new_tokens=32,
machine=machine,
)
finish = time.perf_counter()
outcome[“wall_seconds”] = finish – begin
return outcome
def benchmark_concurrent(mannequin, tokenizer, prompts, machine=“cpu”, employees=4):
outcomes = []
begin = time.perf_counter()
with ThreadPoolExecutor(max_workers=employees) as pool:
futures = [
pool.submit(run_prompt, model, tokenizer, prompt, device)
for prompt in prompts
]
for future in as_completed(futures):
outcomes.append(future.outcome())
finish = time.perf_counter()
total_output_tokens = sum(merchandise[“output_tokens”] for merchandise in outcomes)
return {
“requests”: len(outcomes),
“total_seconds”: finish – begin,
“output_tokens”: total_output_tokens,
“output_tokens_per_second”: total_output_tokens / (finish – begin),
“latency_summary”: summarize([item[“wall_seconds”] for merchandise in outcomes]),
}
This benchmark is for illustration solely. It isn’t a substitute for an actual serving benchmark. It doesn’t mannequin HTTP overhead, streaming, request queues, batching, cancellation, or cache eviction. However it’s a helpful subsequent step after a single-request benchmark the place you’ll be able to run the mannequin in parallel and observe the per-request latency. The Python GIL (International Interpreter Lock) is normally not the primary concern right here as a result of heavyweight mannequin execution is usually offloaded to compiled code. Concurrent use of the identical mannequin or tensors from a number of threads is unsafe with out synchronization, so share one mannequin on CUDA solely with a lock or a single employee thread.
When benchmarking an actual server, document at the least:
Variety of concurrent customers
Immediate token distribution
Output token distribution
Request charge
TTFT (Time to first token) percentiles
Inter-token latency percentiles
Complete tokens per second
Error charge and timeout charge
The distributions matter. A benchmark with all prompts at precisely 128 tokens and all outputs at precisely 128 tokens is simple to match, however it might not characterize your software.
A number of GPUs and A number of Machines
A number of GPUs can be utilized for inference in a number of alternative ways. Completely different approaches can drastically change the efficiency of your system.
The best method is replication. You load one copy of the mannequin on every GPU and route completely different requests to completely different replicas. This will increase throughput and is simple to cause about, however every GPU should have sufficient reminiscence for the total mannequin and its KV cache.
One other method is to separate one mannequin throughout a number of GPUs. Tensor parallelism splits weight matrices throughout units. Pipeline parallelism locations completely different layers on completely different units. Context parallelism partitions sequence work. Knowledgeable parallelism is used for mixture-of-experts fashions. These strategies permit bigger fashions to run, however they introduce communication overhead and might enhance latency.
A number of machines add one other layer. A system might use many replicas throughout machines for top request quantity. It might additionally cut up a single massive mannequin throughout machines, however that is harder as a result of community communication is slower than communication inside one machine. For low-latency serving, crossing machine boundaries inside one ahead move ought to be handled as costly.
Measuring efficiency of a system with a number of GPUs or a number of machines provides a brand new dimension of communication and synchronization overhead. Earlier than selecting a multi-GPU or multi-machine design, reply these questions:
Are you serving one massive mannequin or many smaller fashions?
Are you restricted by mannequin weight reminiscence or KV cache reminiscence?
Do you want decrease latency, larger throughput, or each?
Are requests impartial, or do they share lengthy immediate prefixes?
Can one GPU maintain the mannequin, or should the mannequin be partitioned?
These questions matter as a result of one of the best design will depend on the bottleneck. Including GPUs doesn’t routinely make a single request sooner. It might assist throughput by way of replication, or it might make a bigger mannequin doable by way of partitioning. The benchmark ought to present which impact you might be getting.
Value per Token
Value is a efficiency metric. A sooner system that makes use of way more costly {hardware} is probably not higher for an software.
A easy price estimate is:
cost_per_output_token = hardware_cost_per_second / output_tokens_per_second
cost_per_output_token = hardware_cost_per_second / output_tokens_per_second
If a GPU occasion prices 3 {dollars} per hour and the service generates 1,000 output tokens per second:
hardware_cost_per_second = 3.00 / 3600 = 0.000833
cost_per_output_token = 0.000833 / 1000
= 0.000000833
hardware_cost_per_second = 3.00 / 3600 = 0.000833
cost_per_output_token = 0.000833 / 1000
= 0.000000833
That is lower than one millionth of a greenback per output token for {hardware} alone. An actual calculation may additionally embody idle capability, storage, networking, engineering time, orchestration overhead, and failed requests.
Value ought to be in contrast with high quality. Quantization, smaller fashions, and routing can cut back price, however they could change mannequin habits. An environment friendly inference system just isn’t merely the quickest one. It’s the one which meets high quality and reliability necessities on the lowest sensible price.
Additional Studying
Under are some assets it’s possible you’ll discover helpful:
Little’s legislation, on Wikipedia.It is a helpful queueing-theory outcome for relating common concurrency, arrival charge, and response time. It’s a useful psychological mannequin when reasoning about request charge, latency, and the variety of in-flight inference requests.
Metrics, in NVIDIA NIM LLMs Benchmarking.This web page defines widespread LLM inference metrics equivalent to time to first token, end-to-end latency, inter-token latency, tokens per second, and requests per second.
MLPerf Inference, by MLCommons.It is a broadly used benchmark suite for measuring inference efficiency throughout deployment situations. It isn’t restricted to LLMs, nevertheless it supplies helpful self-discipline round repeatable benchmarking and reporting.
torch.profiler, within the PyTorch documentation.That is the primary PyTorch profiling interface for amassing CPU and accelerator exercise, operator timings, reminiscence data, tensor shapes, and traces that may be inspected later.
NVIDIA Nsight Methods Consumer Information, by NVIDIA.Nsight Methods is beneficial when wall-clock timers usually are not sufficient and also you want a timeline of CUDA API calls, GPU kernels, reminiscence copies, CPU work, and synchronization.
Taming the Titans: A Survey of Environment friendly LLM Inference Serving, by Zhen et al.This survey provides a broader view of LLM inference serving, together with request scheduling, mannequin placement, storage administration, disaggregation, load balancing, and cluster-level serving points.
Abstract
On this chapter, you discovered tips on how to measure LLM inference efficiency. You noticed why prefill and decode ought to be measured individually, tips on how to use wall-clock timers and CUDA occasions, tips on how to document reminiscence utilization, and tips on how to report latency percentiles. You additionally discovered that a number of GPUs can imply replication for extra throughput or partitioning for bigger fashions, and that the benchmark ought to make this distinction clear.
Within the subsequent a part of the e-book, you’ll start finding out strategies for making one mannequin sooner, beginning with floating-point precision.
