A long prompt makes your chat wait before the first word. Why? Size prefill and decode in Python
The tokens per second guide sized how fast a chat streams once it has started, and assumed the prompt was already read. Users also complain about two other things. A long prompt leaves the chat blank for more than a second. And a stream stalls when someone else sends a long prompt. By the end of this guide you can:
- Tell from the prompt length whether the wait before the first token is set by arithmetic or by memory reads, and size it.
- Explain why a long prompt makes other chats stutter, and read the trade that a smaller chunk budget makes.
- Check the idea against a public measured table, and say what 4-bit weights do and do not change.
You need Python 3.12 or newer. No GPU, no API key, no network and no packages to install.
Before you start
You should know Python functions, loops and f-strings, and the cache size, bits per weight and decode ceiling from the three guides we link and do not repeat. If not, open these chapters first. Each link opens a breakdown for that topic with a revision guide and side chats. Sign in to open it.
- How an LLM generates text one token at a time. A reply is one pass over the prompt, then one pass for each new token.
- Compute-bound and memory-bound work in GPUs. Arithmetic and memory reads are two separate limits. The slower one sets the time.
- LLM latency metrics: time to first token and inter-token latency. The wait before the first word and the pace after it are different measures.
- Batching and scheduling in LLM serving. Chats share one step, and the scheduler decides what goes into it.
- Will 32 chats fit on one GPU? Size the KV cache in Python. Where the 56 KiB of cache per token comes from.
- Will 8-bit weights fit 32 chats on one GPU? Size 16, 8 and 4-bit in Python. The bits per weight for Q8_0 and Q4_0.
- You fit 32 chats on one GPU. How fast will each one decode? Estimate it in Python. The guide before this one. It sized the speed after the first token and assumed the prompt was already read.
The question
A team serves a 7 billion parameter chat model on one GPU. Two complaints arrive. A: the first word takes more than a second when the prompt is long. B: 16 streaming chats stall for over a second when one long prompt arrives. For each one, which limit do you hit, arithmetic or memory? What is the first fix to try: a shorter prompt, a cached prefix or a smaller chunk budget? And do 4-bit weights shorten the wait?
The first answers are a formula on a card we chose. The last checks use a public table that others measured on Apple chips. We ran no GPU.
Four words we use
- Prefill
- The pass over the whole prompt. The Cerebras post says prompt processing, known as prefill, processes the input and produces intermediate model state. All prompt tokens go through together, so the work grows with the prompt. In our formula the pass ends with the first token.
- Decode
- Making the reply one token at a time. The Cerebras post says each step means reading model data from memory again, so memory bandwidth matters a lot.
- Time to first token and inter-token latency
- Time to first token (TTFT) is the wait before the first word. Inter-token latency (ITL) is the time between tokens after that. The Redis post says TTFT mostly reflects prefill time plus queuing and network time.
- Chunk budget
- The most tokens an engine puts in one step. In vLLM it is max_num_batched_tokens. If a prompt does not fit in the budget, the engine cuts it into chunks and spreads them over several steps.
Worked example
Find where the wait switches from memory to arithmetic, add up one chat, see what 4-bit changes, cut a long prompt into pieces, then check the idea against a measured table.
One step for the whole prompt: 1,249 ms, and the 16 chats get no token in it
Pieces of 512 prompt tokens: 16 steps of 80 ms, and the 16 chats get a token after each
One engine step with prompt work in itA gap between steps, where every streaming chat gets a token
Find where the wait switches from memory to arithmetic
The model is Qwen2.5-7B-Instruct, with the numbers of the earlier serving guides: 7.61 billion parameters (model card, read on 20 November 2026) and 56 KiB of cache for each token. We chose a card that reads 1,000 GB per second and does 100 trillion operations per second (100 TFLOPS, dense bf16). Both are our choice, not a measurement. A pass over the prompt reads every weight once. In bf16 that is 15.22 GB, which takes 0.015 seconds. The pass also does about 2 operations per parameter for each prompt token, so its arithmetic takes 2 × 7.61 billion × tokens ÷ 100 trillion seconds. The slower of the two sets the time. A 16 or 64 token prompt needs only 0.002 or 0.010 seconds of arithmetic, so the memory read sets the time at 0.015 seconds. At 128 tokens the arithmetic is 0.019 seconds and it sets the time. The two meet at about 100 tokens. From there the wait grows with the prompt: 0.078, 0.312 and 1.247 seconds for 512, 2,048 and 8,192 tokens. The GPUSmith article finds the same kind of crossover for an H100 from datasheet numbers, at about 300 tokens, because an H100 does about three times more arithmetic for each byte it reads than our card. Read the last column as a floor for this card. A real wait also holds attention over the prompt, scheduling, tokenizing, network and queue time, and no card reaches its peak speed.
Add up one chat
Take one chat with an 8,192 token prompt and a 300 token answer. The first token comes after 1.25 seconds. The pass over the prompt already ends with the first token, so we add no extra step. Every later token is a decode step. It reads the 15.22 GB of weights and the 0.47 GB cache of this chat, so it takes 15.7 ms. The other 299 tokens take 4.69 seconds, and the whole answer takes 5.94 seconds. The wait before the first word is 21 percent of that. The tokens per second guide sized the stream. This guide sizes the wait before it.
See what 4-bit changes
Now change the weight format. With the weights and the cache of one 512 token chat, as in the tokens per second guide, the decode ceiling is 65.6, 123.2 and 232.0 tokens per second for bf16, Q8_0 and Q4_0. That is 3.5 times faster for 4-bit. The wait for the 8,192 token prompt stays at 1.25 seconds in all three, because in this formula the arithmetic sets it, and the arithmetic does not depend on how the weights are stored. So 4-bit weights make the stream faster and do not shorten the wait. The formula does not count the work of unpacking 4-bit weights. Step 7 checks that against a measured table.
Let a long prompt arrive while 16 chats stream
Now 16 chats stream, each with 8,192 tokens in its cache. One step takes 22.7 ms, so each chat gets 44.0 tokens per second. A new prompt of 8,192 tokens arrives. If the engine runs all of it in one step, that step takes 1,249 ms, and the 16 chats get one token in that time, which is 0.8 tokens per second each. That is the stutter. Chunked prefill cuts the prompt into pieces and gives each step one piece plus one token for each of the 16 chats. The vLLM docs say decode requests are scheduled first, the rest of the max_num_batched_tokens budget goes to prefill, and a prefill that does not fit is chunked. Read the table. With pieces of 1,024 tokens the 16 chats wait 158 ms between tokens, which is 6.3 tokens per second each, and the new prompt gets its first token after 1.27 seconds. With pieces of 256 tokens it is 41 ms, 24.2 tokens per second and 1.32 seconds. The new prompt pays little at first because these steps are still limited by arithmetic and the arithmetic of the prompt is the same however you cut it. The only extra is the arithmetic of the 16 decode tokens in each step. At 128 tokens a step is limited by the memory read, so the 16 chats are back at 44.0 tokens per second, but the new prompt needs 64 steps that each read the weights again, and its wait grows to 1.46 seconds. So a smaller piece helps the other chats and costs the new prompt more, slowly at first and faster once the step turns memory bound. The vLLM docs say the same in words: smaller values of max_num_batched_tokens give better inter-token latency, and higher values give better time to first token. The first column counts prompt tokens only. The vLLM budget also counts the decode tokens, so 1,024 prompt tokens beside 16 chats is a budget of 1,040.
Reuse a cached prefix
Suppose the start of the prompt matches a prefix the engine already holds, and only the new tokens need the pass. With 0, 50, 90 and 99 percent of the prompt cached, the new tokens are 8,192, 4,096, 819 and 82, and the first token comes after 1.247, 0.623, 0.125 and 0.015 seconds. At 99 percent the memory read sets the time again. This is a best case. It leaves out reading the cached keys and values, the memory the cache takes and a miss. The GPUSmith article notes that new tokens still attend to the cached prefix.
Check it against a measured table: arithmetic follows the cores
The llama.cpp discussion Performance of llama.cpp on Apple Silicon M-series lists measured speeds for a 7B LLaMA model. PP is prompt processing with a batch of 512, and TG is text generation with a batch of 1, so PP is like our prefill and TG is like our decode. We took eight pairs of the same chip and the same memory bandwidth, with fewer and more GPU cores, in F16. Between the two versions of a chip in a pair the number of cores rises 1.14, 1.33, 1.33, 1.19, 1.27, 1.27, 1.33 and 1.25 times. The prompt speed rises 1.15, 1.32, 1.33, 1.23, 1.26, 1.24, 1.37 and 1.22 times. It follows the core count within 3.5 percent. The token speed moves between 0.94 and 1.09 times. This is what the formula says: more arithmetic shortens the prompt pass, and the decode step waits for memory. The plot script in the discussion draws the same pairing, PP against GPU cores and TG against bandwidth.
Check 4-bit against the same table
On the M1 Max with 32 cores the prompt speed is 599.53 tokens per second in F16, 537.37 in Q8_0 and 530.06 in Q4_0. So Q4_0 is 0.88 times F16. The token speed is 23.03, 40.20 and 61.19, so Q4_0 is 2.66 times F16. The 4-bit weights sped up the stream and did not speed up the prompt. We did not test why the prompt speed is a little lower. This is one chip family, one llama.cpp build and one model, and it says nothing about NVIDIA cards or other engines.
Turn it into a first thing to try
The last block of the code is a small function that takes a symptom and prints the likely limit and a first fix. A long wait that grows with the prompt points to the arithmetic on the prompt, so send fewer prompt tokens or reuse a cached prefix. A long wait on a short prompt is not prefill, because a short prompt needs about 15 ms on our card, so look at the queue, tokenizing, the network and cold starts. A stutter when a long prompt arrives points to the prompt sharing each step with the other chats, so lower the chunk budget and measure the stutter and the new prompt's wait again. Slow tokens all the time point to the bytes read in each step, so read fewer bytes. This is a decision aid made from steps 1 to 7. It is not a diagnosis. Measure the time to first token and the time between tokens on your own engine.
Run it yourself
Put the code in one file named first_token.py. Every command here was run on 20 November 2026 with Python 3.12.16 and 3.15.2. The output was the same on both. The code uses only the standard library, so there is nothing to install.
The command uses uv. Without uv, run python first_token.py in Python 3.12 or newer.
first_token.py holds its numbers inside the file and prints one block for each of steps 1 to 8. To size another model, change PARAMETERS and KV_BYTES_PER_TOKEN. To try your own card, change BANDWIDTH and FLOPS. To try another load, change CHATS, CONTEXT, PROMPT and the list of chunk sizes in step 4.
PARAMETERS = 7.61e9 # Qwen2.5-7B-Instruct model card, read on 20 November 2026
KV_BYTES_PER_TOKEN = 2 * 28 * 4 * 128 * 2 # from the KV cache guide: 56 KiB per token
# Chosen for this example. They are not measurements.
BANDWIDTH = 1000e9 # a card that reads 1,000 GB per second
FLOPS = 100e12 # a card that does 100 trillion bf16 operations per second
FORMATS = [("bf16", 16.0), ("Q8_0", 8.5), ("Q4_0", 4.5)] # bits per weight, from the quantization guide
def weights_bytes(bits):
return PARAMETERS * bits / 8
def compute_seconds(tokens):
"""About 2 operations per parameter for each token. Attention over the prompt is left out."""
return 2 * PARAMETERS * tokens / FLOPS
def read_seconds(bits=16.0, chats=0, context=0):
"""Time to read every weight once, plus the cache of every chat in the step."""
return (weights_bytes(bits) + chats * context * KV_BYTES_PER_TOKEN) / BANDWIDTH
def step_seconds(tokens, chats, context, bits=16.0):
"""One engine step takes the slower of its arithmetic and its memory reads."""
return max(compute_seconds(tokens), read_seconds(bits, chats, context))
def first_token_seconds(prompt_tokens, bits=16.0):
"""The pass over the prompt ends with the first token, so no extra decode step is added."""
return step_seconds(prompt_tokens, 0, 0, bits)
print("Step 1 prompt length decides the wait for the first token")
print(f" card: {BANDWIDTH / 1e9:,.0f} GB/s, {FLOPS / 1e12:,.0f} TFLOPS (chosen), bf16 weights {weights_bytes(16.0) / 1e9:.2f} GB")
print(" prompt tokens compute s weight read s first token s bound by")
for tokens in (16, 64, 100, 128, 512, 2048, 8192):
compute, read = compute_seconds(tokens), read_seconds()
print(f" {tokens:>13} {compute:>10.3f} {read:>14.3f} {first_token_seconds(tokens):>14.3f} {'compute' if compute > read else 'memory'}")
print(f" above about {FLOPS * 2 / (2 * BANDWIDTH):.0f} prompt tokens the card is compute bound")
print("Step 2 one chat, 8,192 token prompt, 300 token answer, context held at 8,192")
first = first_token_seconds(8192)
per_token = step_seconds(1, 1, 8192)
rest = 299 * per_token
print(f" first token after {first:.2f} s, then {per_token * 1000:.1f} ms per token, the other 299 tokens take {rest:.2f} s")
print(f" the wait is {first / (first + rest):.0%} of the {first + rest:.2f} s total")
print("Step 3 what 4-bit weights change")
print(" format decode ceiling tokens/s first token for 8,192 tokens s")
for name, bits in FORMATS:
print(f" {name:<6} {1 / read_seconds(bits, 1, 512):>22.1f} {first_token_seconds(8192, bits):>32.2f}")
CHATS, CONTEXT, PROMPT = 16, 8192, 8192
print(f"Step 4 a prompt of {PROMPT:,} tokens arrives while {CHATS} chats stream ({CONTEXT:,} tokens each in cache)")
before = step_seconds(CHATS, CHATS, CONTEXT)
print(f" before: one step takes {before * 1000:.1f} ms, each chat gets {1 / before:.1f} tokens/s")
print(" prompt tokens per step steps new prompt first token s gap for the 16 chats ms tokens/s each")
for chunk in (8192, 4096, 2048, 1024, 512, 256, 128):
step = step_seconds(chunk + CHATS, CHATS, CONTEXT) # the 16 chats add one token each to every step
steps = -(-PROMPT // chunk)
print(f" {chunk:>22} {steps:>6} {steps * step:>25.2f} {step * 1000:>24.0f} {1 / step:>14.1f}")
print("Step 5 a cached prefix skips arithmetic")
print(" cached share new tokens first token s")
for share in (0.0, 0.5, 0.9, 0.99):
new = round(8192 * (1 - share))
print(f" {share:>12.0%} {new:>11} {first_token_seconds(new):>14.3f}")
# llama.cpp discussion 4167, LLaMA 7B, commit 8e672ef: chip, GB/s, fewer cores, PP t/s, TG t/s, more cores, PP t/s, TG t/s (F16)
PAIRS = [
("M1 Pro", 200, 14, 262.65, 12.75, 16, 302.14, 12.75),
("M1 Max", 400, 24, 453.03, 22.55, 32, 599.53, 23.03),
("M1 Ultra", 800, 48, 875.81, 33.92, 64, 1168.89, 37.01),
("M2 Pro", 200, 16, 312.65, 12.47, 19, 384.38, 13.06),
("M2 Max", 400, 30, 600.46, 24.16, 38, 755.67, 24.65),
("M2 Ultra", 800, 60, 1128.59, 39.86, 76, 1401.85, 41.02),
("M3 Ultra", 800, 60, 1121.80, 42.24, 80, 1538.34, 39.78),
("M4 Pro", 273, 16, 381.14, 17.19, 20, 464.48, 17.18),
]
print("Step 6 measured: same bandwidth, more GPU cores (F16, PP = prompt speed, TG = token speed)")
print(" chip GB/s cores cores x prompt x token x")
worst, tg_low, tg_high = 0.0, 9.0, 0.0
for chip, bandwidth, cores_a, pp_a, tg_a, cores_b, pp_b, tg_b in PAIRS:
print(f" {chip:<8} {bandwidth:>5} {cores_a:>3}>{cores_b:<3} {cores_b / cores_a:>8.2f} {pp_b / pp_a:>9.2f} {tg_b / tg_a:>8.2f}")
worst = max(worst, abs((pp_b / pp_a) / (cores_b / cores_a) - 1))
tg_low, tg_high = min(tg_low, tg_b / tg_a), max(tg_high, tg_b / tg_a)
print(f" prompt speed tracks the core count within {worst:.1%}, token speed moves {tg_low:.2f} to {tg_high:.2f} times")
print("Step 7 measured: does 4-bit speed up the prompt? M1 Max, 32 cores")
f16_pp, f16_tg, q8_pp, q8_tg, q4_pp, q4_tg = 599.53, 23.03, 537.37, 40.2, 530.06, 61.19
print(f" prompt speed: F16 {f16_pp}, Q8_0 {q8_pp}, Q4_0 {q4_pp} t/s, Q4_0 is {q4_pp / f16_pp:.2f} times F16")
print(f" token speed: F16 {f16_tg}, Q8_0 {q8_tg}, Q4_0 {q4_tg} t/s, Q4_0 is {q4_tg / f16_tg:.2f} times F16")
FIRST_FIX = {
"long wait, long prompt": (
"arithmetic on the prompt (steps 1 and 2)",
"send fewer prompt tokens or reuse a cached prefix (step 5). 4-bit did not shorten it (steps 3 and 7)",
),
"long wait, short prompt": (
"not prefill: a short prompt needs about 15 ms here (step 1)",
"check queue wait, tokenizing, network and cold starts. The queue guide shows one",
),
"stutter when a long prompt arrives": (
"the long prompt shares each step with the other chats (step 4)",
"lower the chunk budget, then measure the stutter and the new prompt's wait again",
),
"every token is slow": (
"bytes read per step (steps 2 and 3)",
"read fewer bytes: 4-bit weights, shorter chats, fewer chats per step",
),
}
def first_fix(symptom):
limit, fix = FIRST_FIX[symptom]
print(f" {symptom}")
print(f" likely limit: {limit}")
print(f" try first: {fix}")
print("Step 8 from a symptom to a first thing to try (a decision aid, not a diagnosis)")
for symptom in FIRST_FIX:
first_fix(symptom)uv run --no-project --python 3.12 python first_token.pyStep 1 prompt length decides the wait for the first token
card: 1,000 GB/s, 100 TFLOPS (chosen), bf16 weights 15.22 GB
prompt tokens compute s weight read s first token s bound by
16 0.002 0.015 0.015 memory
64 0.010 0.015 0.015 memory
100 0.015 0.015 0.015 memory
128 0.019 0.015 0.019 compute
512 0.078 0.015 0.078 compute
2048 0.312 0.015 0.312 compute
8192 1.247 0.015 1.247 compute
above about 100 prompt tokens the card is compute bound
Step 2 one chat, 8,192 token prompt, 300 token answer, context held at 8,192
first token after 1.25 s, then 15.7 ms per token, the other 299 tokens take 4.69 s
the wait is 21% of the 5.94 s total
Step 3 what 4-bit weights change
format decode ceiling tokens/s first token for 8,192 tokens s
bf16 65.6 1.25
Q8_0 123.2 1.25
Q4_0 232.0 1.25
Step 4 a prompt of 8,192 tokens arrives while 16 chats stream (8,192 tokens each in cache)
before: one step takes 22.7 ms, each chat gets 44.0 tokens/s
prompt tokens per step steps new prompt first token s gap for the 16 chats ms tokens/s each
8192 1 1.25 1249 0.8
4096 2 1.25 626 1.6
2048 4 1.26 314 3.2
1024 8 1.27 158 6.3
512 16 1.29 80 12.4
256 32 1.32 41 24.2
128 64 1.46 23 44.0
Step 5 a cached prefix skips arithmetic
cached share new tokens first token s
0% 8192 1.247
50% 4096 0.623
90% 819 0.125
99% 82 0.015
Step 6 measured: same bandwidth, more GPU cores (F16, PP = prompt speed, TG = token speed)
chip GB/s cores cores x prompt x token x
M1 Pro 200 14>16 1.14 1.15 1.00
M1 Max 400 24>32 1.33 1.32 1.02
M1 Ultra 800 48>64 1.33 1.33 1.09
M2 Pro 200 16>19 1.19 1.23 1.05
M2 Max 400 30>38 1.27 1.26 1.02
M2 Ultra 800 60>76 1.27 1.24 1.03
M3 Ultra 800 60>80 1.33 1.37 0.94
M4 Pro 273 16>20 1.25 1.22 1.00
prompt speed tracks the core count within 3.5%, token speed moves 0.94 to 1.09 times
Step 7 measured: does 4-bit speed up the prompt? M1 Max, 32 cores
prompt speed: F16 599.53, Q8_0 537.37, Q4_0 530.06 t/s, Q4_0 is 0.88 times F16
token speed: F16 23.03, Q8_0 40.2, Q4_0 61.19 t/s, Q4_0 is 2.66 times F16
Step 8 from a symptom to a first thing to try (a decision aid, not a diagnosis)
long wait, long prompt
likely limit: arithmetic on the prompt (steps 1 and 2)
try first: send fewer prompt tokens or reuse a cached prefix (step 5). 4-bit did not shorten it (steps 3 and 7)
long wait, short prompt
likely limit: not prefill: a short prompt needs about 15 ms here (step 1)
try first: check queue wait, tokenizing, network and cold starts. The queue guide shows one
stutter when a long prompt arrives
likely limit: the long prompt shares each step with the other chats (step 4)
try first: lower the chunk budget, then measure the stutter and the new prompt's wait again
every token is slow
likely limit: bytes read per step (steps 2 and 3)
try first: read fewer bytes: 4-bit weights, shorter chats, fewer chats per stepWhere you will see this
The setting is max_num_batched_tokens in the vLLM tuning docs. The words prefill and decode, and the two latency measures, are explained in the Redis post on prefill and decode and the Cerebras post. A longer derivation of the first token floor, with the attention term and a utilization factor, is in the GPUSmith article. The measured numbers of steps 6 and 7 are in the llama.cpp Apple Silicon discussion. What this guide adds is one script with printed output that puts the wait, the other chats and the measured check in one place.
What this does not do
- This is a formula on a chosen card, not a measurement. The 1,000 GB/s and 100 TFLOPS card, the 8,192 token prompts and the 16 chats are our choices. We ran no GPU, no vLLM and no llama.cpp.
- The first token times are a floor. They leave out attention over the prompt. In the GPUSmith article the attention work is 15.4 percent on top of the other arithmetic for a Llama 3 8B prompt of 8,192 tokens, and that is a different model from ours. They also leave out scheduling, tokenizing, network and queue time, and they assume the card reaches its peak speed, which no card does.
- We count 7.61 billion parameters for the arithmetic, as the earlier guides do for the weight bytes. The model card lists 6.53 billion without the embeddings, and an embedding lookup is not a multiplication. So our arithmetic is about 17 percent too high. The GPUSmith article makes the same point about using the nominal size.
- We do not add a decode step after the prompt pass, because the pass ends with the first token. A formula that adds one would add about 15 ms on our card.
- The chunk table is a model of one engine step: the slower of its arithmetic and its memory reads. It leaves out reading the cache of the new prompt and the fixed cost of each step in a real engine, which make small pieces cost more. No engine ran it.
- The cached prefix row is a best case, and a cache hit is not guaranteed.
- The measured rows are llama.cpp on Apple chips with a 7B LLaMA model, not Qwen and not a GPU server. They come from one build, 8e672ef from 21 November 2023, and the discussion says later builds are faster. PP is a batch of 512, so we did not measure a prompt of 8,192 tokens. We left out pairs where the F16 value is missing or the bandwidth differs. We did not check whether other rows, models or engines agree, and we did not test why Q4_0 prompt speed is lower.
- We say nothing about accuracy loss from 4-bit, cost, speculative decoding or running prefill and decode on different hardware. The Cerebras post describes that last design, and we use it only for its definitions, not for its throughput result.
- Docs, model files and tables change. We read every source on the date below.
This is the fourth guide on serving a model. The KV cache guide sizes the cache and the quantization guide sizes the weights. The tokens per second guide sizes the stream, and this one sizes the wait. The next guide asks whether a small draft model can lift the stream, see is speculative decoding worth it. The same stall behind a long job shows up in a queue, see why one slow LLM call stalls your queue. More guides are on the guides page.
Explore this topic in the workspace
Topic Breakdown gives you a structured revision guide for this topic and lets you highlight any phrase to open side-by-side chats with worked examples. Sign in to open it.
Open breakdownOr type How LLM inference works: prefill, decode, KV cache and batching into the topic box on the home page. To see a lesson first, open the sample breakdown on Machine Learning.
Sources
All sources were checked on 20 November 2026. Read each page for the full context.
- llama.cpp discussion 4167: Performance of llama.cpp on Apple Silicon M-series
The measured PP and TG table for a 7B LLaMA model in F16, Q8_0 and Q4_0 at commit 8e672ef, the PP (batch 512) and TG (batch 1) definitions, the llama-bench settings (-p 512 -n 128) and the plot script that charts PP against GPU cores and TG against bandwidth. Read through the GitHub API, last updated 9 November 2026.
- vLLM docs: Optimization and Tuning, Chunked Prefill
Decode requests are scheduled first, a prefill that does not fit in max_num_batched_tokens is chunked, smaller values give better inter-token latency and higher values give better time to first token. Only this section was read.
- Cerebras: Cerebras launches disaggregated inference (17 November 2026)
The definitions of prefill and decode, and that each decode step reads model data from memory again. The first part of the post was read. Its throughput per megawatt result is a vendor measurement on one deployment and is not used.
- Redis: Prefill vs Decode: LLM Inference Phases Explained
The terms TTFT and ITL, that TTFT mostly reflects prefill time plus queuing and network time, and that prefill requests can block decode streams and make streaming users see stutter. Updated 13 November 2026 per the page. The first part was read.
- GPUSmith: Estimating TTFT From FLOPs: The Prefill Compute Floor (28 October 2026)
The floor from FLOPs, the H100 crossover of about 300 tokens (989.5 TFLOPS dense over 3.35 TB/s), the attention term at 15.4 percent for a Llama 3 8B prompt of 8,192 tokens, the point that nominal sizes overstate the arithmetic, and that new tokens still attend to a cached prefix. The first part was read.
- Qwen2.5-7B-Instruct: model card
7.61B parameters, 6.53B without embeddings, 28 layers, and 28 attention heads for Q and 4 for KV.
- Our guide: Will 32 chats fit on one GPU?
56 KiB of cache per token.
- Our guide: 8-bit or 4-bit?
8.5 and 4.5 bits per weight for Q8_0 and Q4_0.
- Our guide: How fast will each chat decode?
The decode ceilings of 65.6, 123.2 and 232.0 tokens per second.