All guides

You fit 32 chats on one GPU. How fast will each one decode? Estimate it in Python

The quantization guide ended with 32 chats of 8,192 tokens fitting on a 24 GiB card with 8-bit weights. Fitting is not the same as being fast. Before you ship it, you want to know how many tokens per second each chat will get, and whether 4-bit makes it faster. By the end of this guide you can:

  • Estimate the tokens per second each chat can get from memory bandwidth, using the bytes that one decode step reads.
  • Explain why, in this formula, 4-bit weights give each chat the same speed once the freed memory is spent on more chats.
  • Check a ceiling against a public measured table, and read your own speed as a share of its ceiling.

You need Python 3.12 or newer. No GPU, no API key, no network and no packages to install.

Before you start

You should know Python functions, lists and f-strings, bytes with GB and GiB, and the cache size and bits per weight from the two guides we link and do not repeat. If not, open these chapters first. Each link opens a breakdown for that topic with a revision guide and side chats. Sign in to open it.

  1. How an LLM generates text one token at a time. Each new token needs one pass through the model, and a reply is many tokens.
  2. Memory bandwidth and compute in GPUs. Bandwidth is how many bytes a chip reads each second. Compute is how many calculations it does.
  3. What the KV cache stores. A decode step reads the cache of every chat as well as the weights.
  4. Batching requests in LLM serving. Chats that share a step share one read of the weights.
  5. Will 32 chats fit on one GPU? Size the KV cache in Python. Where the 56 KiB of cache per token comes from.
  6. Will 8-bit weights fit 32 chats on one GPU? Size 16, 8 and 4-bit in Python. The guide before this one. It gives the bits per weight, and the 16, 32 and 40 chats that fit on a 24 GiB card.

The question

A team serves a 7 billion parameter chat model on one 24 GiB GPU. With 8-bit weights, 32 chats of 8,192 tokens fit. How many tokens per second does each chat get? Does 4-bit make each chat faster? And if you measure a speed, how do you know whether it is good?

The first answers are a formula on a card we chose. The last check uses a public table that others measured on Apple chips. We ran no GPU.

Four words we use

Decode step
One pass through the model that makes the next token for every chat. The Databricks post calls the time to make an output token for each user the time per output token, and says it is how fast each user sees the model.
Memory bandwidth
How many bytes a chip can read from its memory each second, in GB/s. The Databricks post says that at small batch sizes the speed depends on how quickly the model's parameters load from memory, not on how quickly the chip can compute.
Ceiling
The fastest a chat could go if reading the bytes of our formula were its only cost. It is the bandwidth divided by the bytes that one step reads. A real system is usually below it.
Share of ceiling
The measured speed divided by the ceiling. The Databricks post calls this ratio model bandwidth utilization. It defines the achieved bandwidth as the model size plus the KV cache, divided by the time per output token.

Worked example

Find the bytes one step reads, turn them into a speed for each chat, see what 4-bit changes, then check the ceiling against a measured table.

What one step reads when every chat has 8,192 tokens

bf16, 16 chats: 22.74 GB per step, 44.0 tokens per second for each chat

Q4_0, 40 chats: 23.07 GB per step, 43.3 tokens per second for each chat

Weights, read once for all chatsCache of one chat

  1. Find the bytes one step reads

    The model is Qwen2.5-7B-Instruct, with the numbers of the earlier guides: 7.61 billion parameters (model card, read on 17 November 2026) and 56 KiB of cache for each token. We chose a card that reads 1,000 GB per second. In our formula, a decode step reads every weight once and then the cache of each chat, and every chat gets one token from the step. So the tokens per second of one chat is the bandwidth divided by the bytes read in a step. For one chat with 512 tokens, the weights are 15.22 GB in bf16, 8.09 GB in Q8_0 and 4.28 GB in Q4_0, using the bits per weight from the quantization guide. The cache of 512 tokens is about 29 MB. The ceilings are 65.6, 123.2 and 232.0 tokens per second. With one chat and a short context, the weights are almost all of the bytes, so fewer bytes means a faster chat.

  2. Add chats and watch each chat slow down

    Now give every chat 8,192 tokens in bf16. The cache of one chat is 448 MiB, about 0.47 GB. The weights are read once for all the chats, but every chat adds its own cache. One chat reads 15.69 GB in a step and gets 63.7 tokens per second. 4 chats read 17.10 GB and each gets 58.5. 16 chats read 22.74 GB and each gets 44.0. The total goes from 64 to 704, which is 11 times more for 16 times the chats. Each chat is slower and the total is higher. The Databricks post describes the same tradeoff: serving more users at once gives more throughput, and it takes longer to make the tokens of each user.

  3. Spend what 4-bit frees

    Keep 16 chats and the weight format matters a lot. Q8_0 reads 15.60 GB per step and each chat gets 64.1 tokens per second. Q4_0 reads 11.80 GB and each chat gets 84.8. Now spend the freed memory on chats, as the quantization guide did, so that 16, 32 and 40 chats fit. A step then reads 22.74, 23.12 and 23.07 GB, which is almost all of the 23.2 GB of usable memory. Each chat gets 44.0, 43.3 and 43.3 tokens per second. When the card is full, a step reads about all of the memory, so each chat gets about the bandwidth divided by the memory, 1,000 ÷ 23.2 = 43.1 tokens per second in this formula, whatever the weight format. Only the total grows, from 704 to 1,384 and 1,734. So 4-bit can make each chat faster, or it can fit more chats at the same speed. Which one you get depends on what you do with the freed memory.

  4. Spot the common mistake

    Count only the weights and 40 Q4_0 chats look like 234 tokens per second each, which is 1,000 GB/s divided by 4.28 GB. But every step reads the caches too. 40 chats of 8,192 tokens have 18.8 GB of cache, more than 4 times the weights. With the caches, each chat gets 43.3 tokens per second. The weights-only answer is 5.4 times too high, and a team that plans on it will expect a speed far above what the formula allows.

  5. Check the ceiling against a measured table

    The llama.cpp discussion called Performance of llama.cpp on Apple Silicon M-series lists the measured speed of making tokens (TG) for a 7B Llama model in F16, Q8_0 and Q4_0, and the memory bandwidth of each chip. We took the row with the most GPU cores for each of four chips. All four were measured at one llama.cpp commit, 8e672ef. The model is not Qwen. llama-bench prints 6.74 billion parameters for it, so this step uses that number. Our size formula gives 12.55, 6.67 and 3.53 GiB, and llama-bench prints 12.55, 6.67 and mostly 3.56 GiB, so the formula is within 1 percent of the real files. The ceiling is the bandwidth divided by the weight bytes, and the table shows the measured speed as a share of it. On the M2 the share is 90.6, 87.4 and 83.1 percent for F16, Q8_0 and Q4_0. On the M1 Ultra it is 62.4, 53.6 and 39.7 percent. The share falls as the chip gets faster and as the weights get smaller. On the M1 Max, by our formula the Q4_0 weights are 3.56 times smaller than the F16 weights, but the speed is only 2.66 times higher. So a ceiling is a limit and not a forecast. We did not test why the share falls.

  6. Read your own number

    To read a speed you measured, divide it by the ceiling for the bytes one step reads on your setup. Our rule of thumb is to call 70 percent or more close to the ceiling, 40 to 70 percent some distance from it, and below 40 percent far from it. A share above 100 percent means the bytes or the bandwidth are wrong, so check them first. A gap is a reason to look for other costs. The share does not say what they are, and even at 70 percent the ceiling is 43 percent faster than the measured speed. The code reads five cases. A made-up laptop with 90 GB/s and 6 tokens per second is at 25.3 percent. The M1 Max in Q4_0 is at 58.0 percent, the M2 in F16 at 90.6 percent, and the M1 Ultra in Q4_0 at 39.7 percent. The last case is made up: 32 Qwen chats in Q8_0 on the chosen card, at 32 tokens per second each, is at 74.0 percent. The two lines are ours. We chose them after looking at the 12 rows of step 5, so they are not a result. The M1 Ultra row is an ordinary row of a public table and falls just under 40 percent. For another reference, the Databricks post reports a model bandwidth utilization of 60 percent on 2 H100 GPUs and 55 percent on 4 A100-40GB GPUs, at batch size 1 with its TensorRT-LLM server.

  7. Write the rule

    The tokens per second of each chat is about the bandwidth divided by the weights plus the caches of all the chats in the step. A full card gives about the bandwidth divided by the memory. Treat the result as a ceiling in this formula. Then measure a real number on your own engine and card, for the weight format, the number of chats and the context you plan to use, and divide it by the ceiling. The share can differ between formats: on the M1 Max it is 77.6 percent for F16 and 58.0 percent for Q4_0. The two earlier guides say what fits. This guide estimates how fast it can run.

Run it yourself

Put the code in one file named decode_speed.py. Every command here was run on 17 November 2026 with Python 3.12.16 and 3.15.2. The output was the same on both. The code uses only the standard library, so there is nothing to install.

The command uses uv. Without uv, run python decode_speed.py in Python 3.12 or newer.

decode_speed.py holds its numbers inside the file and prints one block for each of steps 1 to 6. Step 7 is the rule, so it has no code. To size another model, change PARAMETERS and KV_BYTES_PER_TOKEN. To read your own speed, change the CASES list at the end. Each case gives the bytes that one step reads. The Llama cases leave out the cache, as step 5 does, and the Qwen case uses step_bytes.

decode_speed.py
PARAMETERS = 7.61e9  # Qwen2.5-7B-Instruct model card, read on 17 November 2026
KV_BYTES_PER_TOKEN = 2 * 28 * 4 * 128 * 2  # from the KV cache guide: 56 KiB per token

# Chosen for this example. They are not measurements.
GPU_GIB, USABLE = 24, 0.90
BANDWIDTH = 1000e9  # a card that reads 1,000 GB per second
TOKENS = 8192  # context of every chat
FORMATS = [("bf16", 16.0), ("Q8_0", 8.5), ("Q4_0", 4.5)]  # bits per weight, from the quantization guide

GIB = 1024**3
BUDGET = GPU_GIB * GIB * USABLE


def weights_bytes(bits, parameters=PARAMETERS):
    return parameters * bits / 8


def step_bytes(bits, chats, tokens):
    """In our formula, one decode step reads every weight once for all chats, and each chat's own cache."""
    return weights_bytes(bits) + chats * tokens * KV_BYTES_PER_TOKEN


def each_chat(bits, chats, tokens, bandwidth=BANDWIDTH):
    """Tokens per second for each chat. Every chat gets one token per step."""
    return bandwidth / step_bytes(bits, chats, tokens)


def max_chats(bits, tokens):
    return int((BUDGET - weights_bytes(bits)) // (KV_BYTES_PER_TOKEN * tokens))


def show(name, bits, chats):
    each = each_chat(bits, chats, TOKENS)
    print(f"        {name:<6} {chats:>5} {step_bytes(bits, chats, TOKENS) / 1e9:>16.2f} {each:>10.1f} {each * chats:>10.0f}")


print("Step 1  one chat, short context: the weights are most of what a step reads")
print(f"        card bandwidth {BANDWIDTH / 1e9:,.0f} GB/s, context 512 tokens")
print("        format  weights GB  tokens/s ceiling")
for name, bits in FORMATS:
    print(f"        {name:<6} {weights_bytes(bits) / 1e9:>10.2f} {each_chat(bits, 1, 512):>17.1f}")

print(f"Step 2  more chats in bf16, {TOKENS:,} tokens each: weights are shared, caches are not")
print("        chats  GB read per step  each chat  all chats")
for chats in (1, 4, 8, 16):
    read = step_bytes(16.0, chats, TOKENS)
    each = each_chat(16.0, chats, TOKENS)
    print(f"        {chats:>5} {read / 1e9:>16.2f} {each:>10.1f} {each * chats:>10.0f}")

print("Step 3  what each format buys")
print("        keep 16 chats:")
print("        format  chats  GB read per step  each chat  all chats")
for name, bits in FORMATS:
    show(name, bits, 16)
print("        or spend the freed memory on as many chats as fit:")
print("        format  chats  GB read per step  each chat  all chats")
for name, bits in FORMATS:
    show(name, bits, max_chats(bits, TOKENS))
print(f"        usable memory is {BUDGET / 1e9:.1f} GB, so bandwidth / memory = {BANDWIDTH / BUDGET:.1f} tokens/s each")

print("Step 4  the mistake: count only the weights")
bits = 4.5
chats = max_chats(bits, TOKENS)
wrong = BANDWIDTH / weights_bytes(bits)
right = each_chat(bits, chats, TOKENS)
print(f"        Q4_0 with {chats} chats: weights only says {wrong:.0f} tokens/s each, with caches it is {right:.1f}, {wrong / right:.1f} times too high")

print("Step 5  a ceiling is not a forecast: llama.cpp measured table, a 7B Llama model")
LLAMA_PARAMETERS = 6.74e9  # llama-bench prints 6.74 B parameters
LLAMA_FORMATS = [("F16", 16.0), ("Q8_0", 8.5), ("Q4_0", 4.5)]
PRINTED_GIB = (12.55, 6.67, 3.56)  # file sizes llama-bench prints in the same discussion
print("        format  size from the formula GiB  size llama-bench prints GiB")
for (name, bits), printed in zip(LLAMA_FORMATS, PRINTED_GIB):
    print(f"        {name:<6} {weights_bytes(bits, LLAMA_PARAMETERS) / GIB:>24.2f} {printed:>27.2f}")
# (chip, GB/s, measured tokens/s in F16, Q8_0 and Q4_0), the row with the most GPU cores for each chip
MEASURED = [
    ("M2", 100, 6.72, 12.21, 21.91),
    ("M1 Pro", 200, 12.75, 22.34, 36.41),
    ("M1 Max", 400, 23.03, 40.20, 61.19),
    ("M1 Ultra", 800, 37.01, 59.87, 83.73),
]
print("        chip      GB/s  format  ceiling  measured  share of ceiling")
for chip, bandwidth, *speeds in MEASURED:
    for (name, bits), measured in zip(LLAMA_FORMATS, speeds):
        ceiling = bandwidth * 1e9 / weights_bytes(bits, LLAMA_PARAMETERS)
        print(f"        {chip:<8} {bandwidth:>5}  {name:<6} {ceiling:>7.1f} {measured:>9.2f} {measured / ceiling:>16.1%}")
smaller = weights_bytes(16.0) / weights_bytes(4.5)
faster = 61.19 / 23.03
print(f"        M1 Max: by our formula the Q4_0 weights are {smaller:.2f} times smaller than F16, but the speed is only {faster:.2f} times higher")

print("Step 6  read your own number")


def verdict(measured, bandwidth_gb_s, bytes_read):
    """The 70 and 40 percent lines are our rule of thumb from the table above, not a result from any source."""
    share = measured / (bandwidth_gb_s * 1e9 / bytes_read)
    if share > 1:
        return share, "above the ceiling, so check the bytes and the bandwidth you used"
    if share >= 0.7:
        return share, "close to the ceiling"
    if share >= 0.4:
        return share, "some distance from the ceiling"
    return share, "far from the ceiling, so look for other costs"


# (label, measured tokens/s, GB/s, bytes that one step reads)
CASES = [
    ("a laptop with 90 GB/s, Q4_0, 6 tokens/s (made up)", 6.0, 90, weights_bytes(4.5, LLAMA_PARAMETERS)),
    ("M1 Max, Q4_0, 61.19 tokens/s on 400 GB/s", 61.19, 400, weights_bytes(4.5, LLAMA_PARAMETERS)),
    ("M2, F16, 6.72 tokens/s on 100 GB/s", 6.72, 100, weights_bytes(16.0, LLAMA_PARAMETERS)),
    ("M1 Ultra, Q4_0, 83.73 tokens/s on 800 GB/s", 83.73, 800, weights_bytes(4.5, LLAMA_PARAMETERS)),
    ("32 Qwen chats, Q8_0, 32 tokens/s each on 1,000 GB/s (made up)", 32.0, 1000, step_bytes(8.5, 32, TOKENS)),
]
for label, measured, bandwidth_gb_s, bytes_read in CASES:
    share, text = verdict(measured, bandwidth_gb_s, bytes_read)
    print(f"        {label}: {share:.1%} of ceiling, {text}")
Terminal
uv run --no-project --python 3.12 python decode_speed.py
Output
Step 1  one chat, short context: the weights are most of what a step reads
        card bandwidth 1,000 GB/s, context 512 tokens
        format  weights GB  tokens/s ceiling
        bf16        15.22              65.6
        Q8_0         8.09             123.2
        Q4_0         4.28             232.0
Step 2  more chats in bf16, 8,192 tokens each: weights are shared, caches are not
        chats  GB read per step  each chat  all chats
            1            15.69       63.7         64
            4            17.10       58.5        234
            8            18.98       52.7        422
           16            22.74       44.0        704
Step 3  what each format buys
        keep 16 chats:
        format  chats  GB read per step  each chat  all chats
        bf16      16            22.74       44.0        704
        Q8_0      16            15.60       64.1       1026
        Q4_0      16            11.80       84.8       1356
        or spend the freed memory on as many chats as fit:
        format  chats  GB read per step  each chat  all chats
        bf16      16            22.74       44.0        704
        Q8_0      32            23.12       43.3       1384
        Q4_0      40            23.07       43.3       1734
        usable memory is 23.2 GB, so bandwidth / memory = 43.1 tokens/s each
Step 4  the mistake: count only the weights
        Q4_0 with 40 chats: weights only says 234 tokens/s each, with caches it is 43.3, 5.4 times too high
Step 5  a ceiling is not a forecast: llama.cpp measured table, a 7B Llama model
        format  size from the formula GiB  size llama-bench prints GiB
        F16                       12.55                       12.55
        Q8_0                       6.67                        6.67
        Q4_0                       3.53                        3.56
        chip      GB/s  format  ceiling  measured  share of ceiling
        M2         100  F16        7.4      6.72            90.6%
        M2         100  Q8_0      14.0     12.21            87.4%
        M2         100  Q4_0      26.4     21.91            83.1%
        M1 Pro     200  F16       14.8     12.75            85.9%
        M1 Pro     200  Q8_0      27.9     22.34            80.0%
        M1 Pro     200  Q4_0      52.8     36.41            69.0%
        M1 Max     400  F16       29.7     23.03            77.6%
        M1 Max     400  Q8_0      55.9     40.20            72.0%
        M1 Max     400  Q4_0     105.5     61.19            58.0%
        M1 Ultra   800  F16       59.3     37.01            62.4%
        M1 Ultra   800  Q8_0     111.7     59.87            53.6%
        M1 Ultra   800  Q4_0     211.0     83.73            39.7%
        M1 Max: by our formula the Q4_0 weights are 3.56 times smaller than F16, but the speed is only 2.66 times higher
Step 6  read your own number
        a laptop with 90 GB/s, Q4_0, 6 tokens/s (made up): 25.3% of ceiling, far from the ceiling, so look for other costs
        M1 Max, Q4_0, 61.19 tokens/s on 400 GB/s: 58.0% of ceiling, some distance from the ceiling
        M2, F16, 6.72 tokens/s on 100 GB/s: 90.6% of ceiling, close to the ceiling
        M1 Ultra, Q4_0, 83.73 tokens/s on 800 GB/s: 39.7% of ceiling, far from the ceiling, so look for other costs
        32 Qwen chats, Q8_0, 32 tokens/s each on 1,000 GB/s (made up): 74.0% of ceiling, close to the ceiling

Where you will see this

The Databricks post on LLM inference performance defines time per output token and throughput, says more users at once raise throughput and slow each user, and defines model bandwidth utilization with the same bytes as step 1. The measured numbers of step 5 are in the llama.cpp Apple Silicon discussion, where llama-bench prints the speed and the file size of each run. The block layouts behind 8.5 and 4.5 bits are in llama.cpp's ggml-common.h.

What this does not do

  • This is a formula on a chosen card, not a measurement. The 1,000 GB/s card, the 24 GiB size, the 90 percent usable share and the 8,192 token chats are our choices. We ran no GPU and no serving engine.
  • The formula counts every chat in every step with its full 8,192 tokens, and treats each step as limited by memory only. A real server batches the chats that are active at that moment. The Databricks post says that past a certain batch size the work becomes compute bound, and then the ceiling here is too high.
  • The formula says a step reads every weight once. A real step can read a little less, for example when an input embedding table is looked up by row, or more. We did not check the layers of Qwen.
  • The measured rows are llama.cpp on Apple chips with a 7B Llama model, not Qwen and not a GPU server. The table is a community summary with one row for each chip and core count, and it lists the bandwidth of each chip. We took the row with the most GPU cores for four chips and left the other chips out. We did not check whether other rows, other models or other engines give the same shares, and we never apply these shares to the Qwen numbers on the chosen card.
  • In step 5 the formula leaves out the cache that the benchmark reads, and it uses 3.53 GiB for Q4_0 where llama-bench mostly prints 3.56 GiB. Both make the ceiling a little high and the share a little low. The benchmark does not measure bytes read, so the 3.56 times smaller weights are our formula's estimate.
  • The 70 and 40 percent lines come from 12 rows and our own judgment. The M1 Ultra Q4_0 row is at 39.7 percent. A low share tells you to look for other costs. It does not tell you what they are.
  • Q8_0 and Q4_0 are llama.cpp formats, and 8.5 and 4.5 bits per weight come from their block layouts in ggml-common.h. Another engine can store the weights in another way and read other bytes.
  • We say nothing about prefill (the pass over the prompt before the first token), speculative decoding, cost per token or how much quality 4-bit loses. The cache is counted at 2 bytes for each value, as in the KV cache guide.
  • Model files, docs and tables change. We read every source on the date below.

This is the third guide on serving memory. The KV cache guide sizes the cache of each chat, and the quantization guide sizes the weights. This one turns both into a speed. It assumes the prompt was already read. The wait before the first word is the next guide, see why the first token is slow. Guessing ahead to get past the ceiling is covered in is speculative decoding worth it. More guides are on the guides page.

Explore this topic in the workspace

Topic Breakdown gives you a structured revision guide for this topic and lets you highlight any phrase to open side-by-side chats with worked examples. Sign in to open it.

Open breakdown

Or type How LLM inference speed is limited by memory bandwidth into the topic box on the home page. To see a lesson first, open the sample breakdown on Machine Learning.

Sources

All sources were checked on 17 November 2026. Read each page for the full context.

  • llama.cpp discussion 4167: Performance of llama.cpp on Apple Silicon M-series

    The summary table of measured token generation speed (TG) and memory bandwidth for a 7B Llama model in F16, Q8_0 and Q4_0, at commit 8e672ef. Its comments hold llama-bench lines that print 6.74 B parameters and sizes of 12.55 GiB (F16), 6.67 GiB (Q8_0) and 3.56 GiB (Q4_0, 3.53 GiB in some runs). GitHub showed the last update on 9 November 2026, and the page keeps getting new rows.

  • Databricks: LLM Inference Performance Engineering: Best Practices (12 October 2023)

    Defines time per output token and throughput, and says that serving more users at once raises throughput and lengthens each user's time per token. Defines model bandwidth utilization as achieved bandwidth, (model size + KV cache size) divided by the time per output token, over peak bandwidth. Reports 60 percent on 2 H100 GPUs and 55 percent on 4 A100-40GB GPUs at batch size 1, and says that past a certain batch size the work is compute bound. The results are Databricks' own.

  • llama.cpp: ggml-common.h (commit 85c5522)

    block_q8_0 is a 2 byte scale and 32 int8 values, and block_q4_0 is a 2 byte scale and 16 bytes for 32 values of 4 bits, which gives 8.5 and 4.5 bits per weight. The link is pinned to a commit, so the file can differ on the main branch.

  • Qwen2.5-7B-Instruct: model card

    7.61B parameters, 28 layers, and 28 attention heads for Q and 4 for KV.

  • Qwen2.5-7B-Instruct: config.json

    hidden_size 3584, num_attention_heads 28, num_hidden_layers 28, num_key_value_heads 4 and torch_dtype bfloat16, which give the 56 KiB of cache per token that the KV cache guide works out.

  • Our guide: Will 32 chats fit on one GPU?

    56 KiB of cache per token and 448 MiB for a chat of 8,192 tokens.

  • Our guide: 8-bit or 4-bit?

    8.5 and 4.5 bits per weight, the 24 GiB card with 90 percent usable, and 16, 32 and 40 chats at 8,192 tokens for bf16, Q8_0 and Q4_0.