All guides

Will 32 chats fit on one GPU? Size the KV cache in Python

A model that serves one chat well can run out of GPU memory as you add chats. The model is not too big for the card. Every open chat keeps its own cache, and the caches add up. If you serve a chat model yourself, you need the number before the memory error shows it to you. By the end of this guide you can:

  • Work out how many bytes of KV cache one token needs, from a model's config file.
  • Add the weights and the cache together to say whether a number of chats fits on one GPU.
  • Choose a context limit and a chat cap, and spot the mistake of counting every attention head.

You need Python 3.12 or newer. No GPU, no API key, no network and no packages to install.

Before you start

You should know Python functions, dictionaries and reading JSON, bytes with KiB, MiB and GiB, and that a chat model has a fixed number of parameters, each stored as a number of a set size. If not, open these chapters first. Each link opens a breakdown for that topic with a revision guide and side chats. Sign in to open it.

  1. Attention in transformers. Each new token looks back at the tokens before it.
  2. What the KV cache stores. The model keeps the keys and values of earlier tokens so it does not work them out again.
  3. Grouped-query attention. Why a model has fewer key-value heads than query heads.
  4. Context length and memory. More tokens in a chat means more memory.
  5. Batching and concurrency in LLM serving. A server answers many chats at once.
  6. One slow LLM call is blocking your queue. Measure it and fix it in Python. Optional, the guide before this one. It shows that more workers stop helping at some point. This guide shows what memory allows.

The question

A team serves a 7 billion parameter chat model on one GPU. One chat works. Now they want 32 chats open at once, each up to 8,192 tokens long. Will that fit on a 24 GiB card? If not, what should they change?

The answer below is a formula, not a measurement. We chose the 24 GiB card, the 90 percent usable share and the number of chats. We did not run the model or any GPU.

Four words we use

KV cache
Memory that holds the keys and values the model already worked out for earlier tokens, so it can reuse them. The Transformers docs describe it this way. Every chat has its own tokens, so every chat has its own cache.
Token
A piece of text that the model reads, often a word or part of one. The context length in a config file is counted in tokens.
Key-value head
A model has query heads and key-value heads. Grouped-query attention gives a model fewer key-value heads than query heads, so several query heads share one. Qwen2.5-7B has 28 query heads and 4 key-value heads, so 7 query heads share each one. Only the key-value heads are stored in the cache.
Context limit and chat cap
These are our words. The context limit is the most tokens one chat may hold. The chat cap is the most chats open at once. Servers name these settings in their own way.

Worked example

Work out the cache of one token, then of one chat, then of many chats, and compare it with what is left on the card.

  1. State the question and the inputs

    The model is Qwen2.5-7B-Instruct. Its config.json, read on 14 November 2026, gives a hidden size of 3,584, 28 layers, 28 attention heads, 4 key-value heads, a limit of 32,768 positions and bfloat16 numbers. The model card gives 7.61 billion parameters. We chose a 24 GiB GPU with 90 percent usable, and a goal of 32 chats at 8,192 tokens each. Every example stays within 32,768 tokens, because the model card says the current config.json is set for context up to 32,768 tokens and that longer inputs need an extra setting that we did not use.

  2. Find the cache for one token

    The config has no head size, so the code divides 3,584 by 28 heads and gets 128. The Transformers code for this model does the same when a config has no head size. A token needs one key and one value for every layer and every key-value head. That is 2 × 28 layers × 4 key-value heads × 128 × 2 bytes, which is 57,344 bytes or 56 KiB. The last 2 is the size of one number. The config says bfloat16, which PyTorch lists as a 16-bit type, so 2 bytes. We assume the cache uses the same type as the weights.

  3. Add up one chat

    At 1,024 tokens the cache of one chat is 56 MiB. At 8,192 tokens it is 448 MiB, and at 32,768 tokens it is 1,792 MiB. The cache grows in a straight line with the tokens in the chat. One chat of 8,192 tokens is small next to a 24 GiB card, which is why one chat works.

  4. Add the weights and many chats

    The weights take 7.61 billion × 2 bytes, which is 14.17 GiB. A 24 GiB card with 90 percent usable gives 21.6 GiB, so 7.43 GiB is left for the cache. At 8,192 tokens each, 8 chats need 3.5 GiB and 16 chats need 7.0 GiB, and both fit. 32 chats need 14.0 GiB, so they do not fit. Weights plus cache would be 28.2 GiB against 21.6 GiB usable. The count of 16 is close. 17 chats would need 7.44 GiB, just over the 7.43 GiB left, so a small change to the usable share changes the answer.

  5. Trade context for chats

    The same 7.43 GiB holds 67 chats at 2,048 tokens, 33 at 4,096, 16 at 8,192, 8 at 16,384 and 4 at 32,768. Each time the limit doubles, about half as many chats fit. 32 chats at 4,096 tokens need 7.0 GiB, which fits. So the 32 chats can work if each chat is limited to 4,096 tokens. The context limit is the part you can change without a bigger card.

  6. Spot the common mistake

    Count all 28 attention heads instead of the 4 key-value heads and the cache is 392 KiB per token. That is 7 times too big, because 28 divided by 4 is 7. It says only 2 chats fit at 8,192 tokens, when the right answer is 16. A team that trusts that number may buy a bigger card, or cut the chat cap, when it did not need to. The GQA paper describes grouped-query attention as using more than one key-value head and fewer than the number of query heads. Read the number of key-value heads from the config of your own model.

  7. Write the rule

    Pick the chat cap first, here 32. The most tokens each chat may use is the budget divided by the cache per token and by the cap. Here that is 4,344 tokens, so set the limit to 4,096. Or pick the limit your chats really need and divide the other way, as in step 5. Then set the cap you promise below the number you computed. The sum counts every chat at its full limit, and an engine also needs memory for other things. The vLLM docs say to limit the context length (max_model_len) and the maximum batch size (max_num_seqs) to use less memory, and that its CUDA graphs take extra GPU memory. We did not run vLLM, so test your own setup under load before you trust a number.

Run it yourself

Put the code in one file named kv_cache.py. Every command here was run on 14 November 2026 with Python 3.12.16 and 3.15.2. The output was the same on both. The code uses only the standard library, so there is nothing to install.

The command uses uv. Without uv, run python kv_cache.py in Python 3.12 or newer.

kv_cache.py holds the config numbers inside the file and prints one block for each of steps 1 to 7. To size another model, replace the config numbers and the parameter count with those of your model.

kv_cache.py
import json

# A subset of config.json from Qwen/Qwen2.5-7B-Instruct, read on 14 November 2026.
CONFIG = json.loads("""
{
  "hidden_size": 3584,
  "num_hidden_layers": 28,
  "num_attention_heads": 28,
  "num_key_value_heads": 4,
  "max_position_embeddings": 32768,
  "torch_dtype": "bfloat16"
}
""")
PARAMETERS = 7.61e9  # from the model card, the same day
BYTES = {"bfloat16": 2, "float16": 2, "float32": 4}[CONFIG["torch_dtype"]]

# Chosen for this example. They are not measurements.
GPU_GIB = 24
USABLE = 0.90
CHATS, TOKENS = 32, 8192

KIB, MIB, GIB = 1024, 1024**2, 1024**3


def head_size(config):
    # config.json has no head size, so divide the hidden size by the attention heads.
    return config["hidden_size"] // config["num_attention_heads"]


def kv_bytes_per_token(config, kv_heads=None):
    """Bytes of cache for one token: keys and values (2) for every layer, key-value head and head size."""
    kv_heads = config["num_key_value_heads"] if kv_heads is None else kv_heads
    return 2 * config["num_hidden_layers"] * kv_heads * head_size(config) * BYTES


def max_chats(budget, per_token, context):
    """Most chats that fit when every chat uses its full context limit."""
    return int(budget // (per_token * context))


def plural(n, word):
    return f"{n} {word}" if n == 1 else f"{n} {word}s"


per_token = kv_bytes_per_token(CONFIG)
weights = PARAMETERS * BYTES
budget = GPU_GIB * GIB * USABLE - weights

print("Step 1  the question")
print(f"        Qwen2.5-7B-Instruct: {PARAMETERS / 1e9:.2f}B parameters, {CONFIG['torch_dtype']} ({BYTES} bytes each)")
print(f"        one {GPU_GIB} GiB GPU, {USABLE:.0%} usable")
print(f"        goal: {CHATS} chats at {TOKENS:,} tokens each")

print("Step 2  cache bytes for one token")
print(f"        head size: {CONFIG['hidden_size']} / {CONFIG['num_attention_heads']} = {head_size(CONFIG)}")
print(
    f"        2 x {CONFIG['num_hidden_layers']} layers x {CONFIG['num_key_value_heads']} key-value heads"
    f" x {head_size(CONFIG)} x {BYTES} bytes = {per_token:,} bytes ({per_token // KIB} KiB)"
)

print("Step 3  the cache of one chat")
for tokens in (1024, 8192, 32768):
    print(f"        {tokens:>6,} tokens: {per_token * tokens / MIB:>5,.0f} MiB")

print(f"Step 4  weights, budget and many chats at {TOKENS:,} tokens each")
print(f"        weights: {PARAMETERS / 1e9:.2f}B x {BYTES} bytes = {weights / GIB:.2f} GiB")
print(f"        budget for the cache: {GPU_GIB} GiB x {USABLE:.0%} - {weights / GIB:.2f} GiB = {budget / GIB:.2f} GiB")
print("        chats  cache (GiB)  fits")
for chats in (1, 8, 16, 32):
    cache = per_token * TOKENS * chats
    print(f"        {chats:>5}  {cache / GIB:>11.1f}  {'yes' if cache <= budget else 'no'}")

print(f"Step 5  context limit and the most chats that fit in {budget / GIB:.2f} GiB")
print("        context limit  most chats")
for context in (2048, 4096, 8192, 16384, 32768):
    print(f"        {context:>13,}  {max_chats(budget, per_token, context):>10}")
need = per_token * 4096 * CHATS
print(f"        {plural(CHATS, 'chat')} at 4,096 tokens need {need / GIB:.1f} GiB, which {'fits' if need <= budget else 'does not fit'}")

print("Step 6  the mistake: count every attention head instead of the key-value heads")
wrong = kv_bytes_per_token(CONFIG, kv_heads=CONFIG["num_attention_heads"])
print(f"        cache per token: {wrong // KIB} KiB, right is {per_token // KIB} KiB, {wrong // per_token} times too big")
print(f"        chats at {TOKENS:,} tokens: {max_chats(budget, wrong, TOKENS)}, right is {max_chats(budget, per_token, TOKENS)}")

print("Step 7  the rule: pick the chat cap, then the context limit")
longest = int(budget // (per_token * CHATS))
limit = max(c for c in (1024, 2048, 4096, 8192, 16384, 32768) if c <= longest)
print(f"        {plural(CHATS, 'chat')} can each use at most {longest:,} tokens, so set the limit to {limit:,}")
Terminal
uv run --no-project --python 3.12 python kv_cache.py
Output
Step 1  the question
        Qwen2.5-7B-Instruct: 7.61B parameters, bfloat16 (2 bytes each)
        one 24 GiB GPU, 90% usable
        goal: 32 chats at 8,192 tokens each
Step 2  cache bytes for one token
        head size: 3584 / 28 = 128
        2 x 28 layers x 4 key-value heads x 128 x 2 bytes = 57,344 bytes (56 KiB)
Step 3  the cache of one chat
         1,024 tokens:    56 MiB
         8,192 tokens:   448 MiB
        32,768 tokens: 1,792 MiB
Step 4  weights, budget and many chats at 8,192 tokens each
        weights: 7.61B x 2 bytes = 14.17 GiB
        budget for the cache: 24 GiB x 90% - 14.17 GiB = 7.43 GiB
        chats  cache (GiB)  fits
            1          0.4  yes
            8          3.5  yes
           16          7.0  yes
           32         14.0  no
Step 5  context limit and the most chats that fit in 7.43 GiB
        context limit  most chats
                2,048          67
                4,096          33
                8,192          16
               16,384           8
               32,768           4
        32 chats at 4,096 tokens need 7.0 GiB, which fits
Step 6  the mistake: count every attention head instead of the key-value heads
        cache per token: 392 KiB, right is 56 KiB, 7 times too big
        chats at 8,192 tokens: 2, right is 16
Step 7  the rule: pick the chat cap, then the context limit
        32 chats can each use at most 4,344 tokens, so set the limit to 4,096

Where you will see this

The numbers in step 2 come from the model's own config.json and model card. Both name the 4 key-value heads. The GQA paper is where grouped-query attention comes from.

The vLLM memory page lists the same levers as step 7. You can limit the context length with max_model_len and the maximum batch size with max_num_seqs, use a quantized model, or reduce CUDA graphs, which take extra memory by default. We did not run vLLM, so we did not check how its settings line up with the chat cap in this guide.

The Transformers cache docs say a fixed-size cache reserves a maximum size up front, and that with one very large sequence and many short ones, a lot of the reserved space goes to waste. Counting every chat at its full limit, as we do, reserves the maximum for each chat in the same way.

What this does not do

  • This is a formula, not a measurement. We did not run a model on a GPU, and nothing here measures any GPU, model or engine. The 24 GiB card and the 90 percent usable share are our choices.
  • The 90 percent is a guess at the memory an engine keeps for other things. The vLLM page says CUDA graphs take extra GPU memory, and we did not size that or anything else.
  • We assume the cache uses the same 2 byte type as the weights. An engine can store it another way.
  • The sum counts every chat at its full context limit, which is the worst case. Real chats are often shorter, so more may fit, but a few long chats can fill the budget.
  • An engine can store the cache in other ways, such as in blocks, or share parts of it between chats. We did not check any engine, so real numbers can differ from these.
  • The numbers belong to this one model. Another model has other numbers, so read the config of yours. Some models use sliding window attention, and the Transformers docs say the cache of those layers stops growing at the window. This model's config sets use_sliding_window to false, so we count every token in every layer.
  • We kept every example within 32,768 tokens. The model card describes longer inputs with an extra setting, YaRN, and we did not use it, so nothing here claims anything above 32,768 tokens.
  • How many chats fit is not how fast they run. More chats at once can slow each reply. We did not measure speed. A later guide estimates it from memory bandwidth.
  • We did not size 8-bit or 4-bit weights in this guide. The vLLM page says quantized models take less memory at the cost of lower precision. The next guide sizes them.
  • Model files can change. Check the config of the model version you serve.

Next, size 8-bit and 4-bit weights for the same card in Will 8-bit weights fit 32 chats on one GPU? Size 16, 8 and 4-bit in Python, then see how fast each chat runs in You fit 32 chats on one GPU. How fast will each one decode? Estimate it in Python. The guide before this one shows why one slow LLM call stalls your queue, where more workers stop helping at some point. This guide shows the other limit, memory. More guides are on the guides page.

Explore this topic in the workspace

Topic Breakdown gives you a structured revision guide for this topic and lets you highlight any phrase to open side-by-side chats with worked examples. Sign in to open it.

Open breakdown

Or type KV cache memory and concurrency for LLM serving into the topic box on the home page. To see a lesson first, open the sample breakdown on Machine Learning.

Sources

All sources were checked on 14 November 2026. Read each page for the full context.

  • Qwen2.5-7B-Instruct: config.json

    The numbers in the code: hidden_size 3584, num_hidden_layers 28, num_attention_heads 28, num_key_value_heads 4, max_position_embeddings 32768, torch_dtype bfloat16 and use_sliding_window false. The file was read on the date above and can change.

  • Qwen2.5-7B-Instruct: model card

    7.61B parameters, 28 layers, and 28 attention heads for Q and 4 for KV. It says the current config.json is set for context length up to 32,768 tokens, and that longer inputs use YaRN.

  • Transformers source: modeling_qwen2.py (main branch)

    The attention code takes the head size from the config when it has one, and otherwise divides the hidden size by the number of attention heads. Its key and value layers are sized by the number of key-value heads times the head size. This is the main branch, which changes.

  • Transformers docs: Cache strategies

    A KV cache stores key-value calculations so they can be reused without recomputing them. A fixed-size cache pre-allocates a maximum size, and the page says a lot is wasted on the short sequences when there is one very large one. For sliding window layers, the cache stops growing at the window size.

  • vLLM docs: Conserving Memory (stable)

    You can reduce memory use by limiting the context length (max_model_len) and the maximum batch size (max_num_seqs). Quantized models take less memory at the cost of lower precision. CUDA graphs take up extra memory in the GPU by default. The page footer shows the date October 9, 2026.

  • GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (arXiv 2305.13245)

    The abstract says multi-query attention uses a single key-value head, and grouped-query attention uses an intermediate number, more than one and less than the number of query heads.

  • PyTorch docs: Tensor Attributes

    torch.bfloat16 is a 16-bit floating point type, so each number takes 2 bytes. The page redirected to the PyTorch 2.15 docs.