Will 8-bit weights fit 32 chats on one GPU? Size 16, 8 and 4-bit in Python
The KV cache guide ended with a 24 GiB card that holds 16 chats of 8,192 tokens, and 32 did not fit. Someone will say "just use 4-bit". That can help, but you should know what it saves, what it does not save and what can go wrong before you try it. By the end of this guide you can:
- Work out the bits per weight of a weight format from its block layout, and see why 4-bit weights cost 4.5 bits, not 4.
- Turn bits per weight into the memory the weights take and the number of chats that fit on one GPU, at 16, 8 and 4 bits.
- Reproduce why one big value can ruin 4-bit weights that share one scale, and what a scale for each block changes.
You need Python 3.12 or newer. No GPU, no API key, no network and no packages to install.
Before you start
You should know Python functions, lists and f-strings, bytes with KiB and GiB, and the cache sizes from the KV cache guide, which we link and do not repeat. If not, open these chapters first. Each link opens a breakdown for that topic with a revision guide and side chats. Sign in to open it.
- Number formats: bfloat16, int8 and int4. How many bits a number takes, and what a smaller number type can hold.
- Weight quantization in neural networks. Storing the weights of a model in fewer bits so it needs less memory.
- Quantization scales and outliers. Why a group of weights shares a scale, and why one big value can spoil it.
- What the KV cache stores. The cache is memory on top of the weights, and every chat has its own.
- Evaluating a model before release. Bits decide memory. Only a test of your own cases decides quality.
- Will 32 chats fit on one GPU? Size the KV cache in Python. The guide before this one. This guide starts from its answer, 16 chats on a 24 GiB card, and reuses its cache size of 56 KiB per token.
The question
A team serves a 7 billion parameter chat model on one 24 GiB GPU. With 16 bit weights, 16 chats of 8,192 tokens fit and 32 do not. A teammate says to store the weights in 8 bits or 4 bits. Does either make the 32 chats fit, by how much, and what does each cost?
The answer below is a formula and a simulation on made-up weights. We chose the 24 GiB card, the 90 percent usable share and the 32 chats. We did not load a model or use a GPU.
Four words we use
- Quantization
- Storing numbers in fewer bits. Here it means the weights of a model. The Transformers docs say quantization stores a model's weights in fewer bits so the model needs less memory, and that fewer bits cost some accuracy.
- Block and scale
- A block is a small group of weights that share one scale. To store a weight we divide it by the scale, round it to a whole number and keep that number. To use it again we multiply by the scale. In llama.cpp's Q8_0 and Q4_0 formats a block is 32 weights and its scale takes 2 bytes.
- Bits per weight
- The bits in a block divided by the weights in it. The scale counts too, so Q4_0 is 4.5 bits per weight and not 4.
- Big value
- A weight much larger than the others in its group. These are our words. The LLM.int8() paper uses the word outlier for large features in real models, which is a related idea and not the same thing.
Worked example
Work out the bits of each format, the memory its weights take and the chats that fit, then see why 4-bit weights need a scale for each block.
Recap the problem
The KV cache guide found 57,344 bytes of cache for one token of Qwen2.5-7B-Instruct, which is 56 KiB. We reuse that number. The model card, read on 15 November 2026, gives 7.61 billion parameters. 32 chats at 8,192 tokens need 14.0 GiB of cache. We chose a 24 GiB card with 90 percent usable, which is 21.6 GiB. The weights at 16 bits are 7.61 billion × 2 bytes, 14.17 GiB, and they leave 7.43 GiB. So 16 chats fit and 32 do not. That is where the KV cache guide stopped.
Find the bits per weight from the block layout
A weight format is a block layout. In llama.cpp's ggml-common.h, read on 15 November 2026 at a pinned commit, block_q8_0 is a scale of 2 bytes and 32 integers of 8 bits. That is 34 bytes for 32 weights, so 34 × 8 / 32 = 8.5 bits per weight. block_q4_0 is a scale of 2 bytes and 16 bytes that hold 32 integers of 4 bits, two in each byte. That is 18 bytes for 32 weights, which is 4.5 bits per weight. The scale is the extra half bit. bf16 has no scale and takes 16 bits. Q8_0 and Q4_0 are llama.cpp's formats. Another format can store its scales in another way, so 4.5 is not a rule for every 4-bit format.
Turn bits into weights, space left and chats
The weights take 7.61 billion × bits / 8 bytes. That is 14.17 GiB at 16 bits, 7.53 GiB at Q8_0 and 3.99 GiB at Q4_0. The usable part of the card is 21.6 GiB, so the space left for the cache is 7.43, 14.07 and 17.61 GiB. One chat of 8,192 tokens needs 448 MiB of cache, from the KV cache guide, and we still count the cache at 2 bytes for each value. The most chats at 8,192 tokens is then 16, 32 and 40. At 4,096 tokens it is 33, 64 and 80. Only the weights change between the three. The cache of one chat does not.
Spot the common mistake
Count 4.0 bits and forget the scale, and the weights look like 3.54 GiB. That leaves 18.06 GiB, and the code says 41 chats fit. The right answer is 40, because the scale makes it 4.5 bits. One chat is a small miss, but at the edge it is the chat that does not fit. The cache is still counted at 2 bytes, because this guide changes only the weights.
Answer the question
The goal is 32 chats at 8,192 tokens, which is 14.0 GiB of cache. 16 bits leaves 7.43 GiB, so no. Q8_0 leaves 14.07 GiB, so yes, with 0.07 GiB to spare. Q4_0 leaves 17.61 GiB, with 3.61 GiB to spare, and 40 chats fit. So 8-bit weights make the 32 chats fit on paper, but only just, and 4-bit leaves room. Memory is not the only thing to weigh. The Transformers overview says fewer bits cost some accuracy, and we did not measure that.
Why every block has its own scale
Build a row of 4,096 made-up weights with random.Random(7).gauss(0, 0.02). Put 0.6 in position 1,000, which is 30 times the usual size. The code quantizes the row two ways: one scale for the whole row, or one scale for each block of 32 weights. The scale is the biggest absolute value in the group divided by the biggest whole number the bits allow, 127 at 8 bits and 7 at 4 bits. Each weight is rounded to a whole number of steps and multiplied back, so a weight smaller than half a step comes back as zero. The table shows the relative error, where 0 is perfect and 1 means the error is as big as the weights, and the share of weights that come back as zero. At 4 bits with the big value, one scale for the whole row gives an error of 0.862 and 96.8 percent of the weights come back as zero. One step is 0.6 / 7, about 0.086, so almost every weight is smaller than half a step. One scale for each 32 weights gives 0.117 and 13.5 percent, because only the 32 weights in the block of the big value share its scale. The other blocks work as before, and 13.5 percent is close to the 13.1 percent of the same row without the big value.
Read the table with care
Without the big value, 4 bits with one scale for the row has an error of 0.142 and 19.4 percent zeros, against 0.095 and 13.1 percent for each 32 weights. So the block scale helps a little when the weights are alike and a lot when one is not. The price is the half bit from step 2. A scale for the whole row costs almost nothing, but one big value can spoil the whole row. At 8 bits the same big value costs 0.062 and 9.6 percent with one scale for the row, and 0.007 and 0.8 percent for each 32 weights. The row is made up and the big value is ours. The LLM.int8() abstract says transformer language models have highly systematic emergent features that dominate attention and predictions, and that its method keeps the outlier dimensions in 16 bits while more than 99.9 percent of values are multiplied in 8 bits. So very large values do show up in real models, and this method works around them. That paper is about features of the model and not about the weights in our row, so we do not say our row looks like a real one.
Write the rule
Memory is a formula you can compute: bits per weight from the block layout, then weights, space left and chats. Pick the format that fits with a margin, and check the size of the file you plan to load. The Transformers overview says most GGUF models are dequantized to full precision as they load in Transformers, so they use as much memory as an unquantized model. Quality is a test you run. Bits tell you what fits. They do not tell you whether the model still answers well. Before you ship a quantized model, run your own cases on it and count enough runs, as the guides in the last section show.
Run it yourself
Put the code in one file named weight_formats.py. Every command here was run on 15 November 2026 with Python 3.12.16 and 3.15.2. The output was the same on both. The code uses only the standard library, so there is nothing to install.
The command uses uv. Without uv, run python weight_formats.py in Python 3.12 or newer.
weight_formats.py holds its numbers inside the file and prints one block for each of steps 1 to 6. Steps 7 and 8 are reading and the rule, so they have no code. To size another model, change PARAMETERS and KV_BYTES_PER_TOKEN.
import random
PARAMETERS = 7.61e9 # Qwen2.5-7B-Instruct model card, read on 15 November 2026
KV_BYTES_PER_TOKEN = 2 * 28 * 4 * 128 * 2 # from the KV cache guide: 56 KiB per token
# Chosen for this example. They are not measurements.
GPU_GIB, USABLE = 24, 0.90
CHATS, TOKENS = 32, 8192
GIB = 1024**3
BUDGET = GPU_GIB * GIB * USABLE
# (name, bytes in one block, weights in one block). The two block layouts are llama.cpp's
# block_q8_0 and block_q4_0 in ggml-common.h: a 2 byte scale, then 32 values of 8 or 4 bits.
FORMATS = [("bf16", 2, 1), ("Q8_0", 2 + 32, 32), ("Q4_0", 2 + 32 // 2, 32)]
def bits_per_weight(block_bytes, weights_per_block):
return block_bytes * 8 / weights_per_block
def weights_bytes(bits):
return PARAMETERS * bits / 8
def max_chats(bits, tokens):
"""Most chats that fit when every chat uses its full context limit."""
return int((BUDGET - weights_bytes(bits)) // (KV_BYTES_PER_TOKEN * tokens))
def quantize(values, bits, block_size):
"""Round to the nearest step. Every block of block_size values shares one scale."""
top = 2 ** (bits - 1)
restored = []
for start in range(0, len(values), block_size):
block = values[start:start + block_size]
scale = max(abs(v) for v in block) / (top - 1)
for v in block:
restored.append(max(-top, min(top - 1, round(v / scale))) * scale)
return restored
def relative_error(original, restored):
squared_error = sum((a - b) ** 2 for a, b in zip(original, restored))
return (squared_error / sum(a * a for a in original)) ** 0.5
need = KV_BYTES_PER_TOKEN * TOKENS * CHATS
print("Step 1 the problem from the KV cache guide")
print(f" {CHATS} chats at {TOKENS:,} tokens need {need / GIB:.1f} GiB of cache")
print(f" bf16 weights leave {(BUDGET - weights_bytes(16)) / GIB:.2f} GiB, so {max_chats(16, TOKENS)} chats fit")
print("Step 2 bits per weight from the block layout")
for name, block_bytes, weights in FORMATS:
print(f" {name:<5} {block_bytes:>2} bytes per block of {weights:>2} = {bits_per_weight(block_bytes, weights):>4.1f} bits per weight")
print(f"Step 3 weights, space left and chats on one {GPU_GIB} GiB GPU")
print(" format bits/weight weights GiB left GiB chats@8,192 chats@4,096")
for name, block_bytes, weights in FORMATS:
bits = bits_per_weight(block_bytes, weights)
left = BUDGET - weights_bytes(bits)
print(f" {name:<6} {bits:>11.2f} {weights_bytes(bits) / GIB:>12.2f} {left / GIB:>9.2f} {max_chats(bits, 8192):>12} {max_chats(bits, 4096):>12}")
print("Step 4 the mistake: count 4.0 bits and forget the scale")
print(f" weights {weights_bytes(4.0) / GIB:.2f} GiB, left {(BUDGET - weights_bytes(4.0)) / GIB:.2f} GiB, chats@8,192: {max_chats(4.0, TOKENS)}, right is {max_chats(4.5, TOKENS)}")
print(f"Step 5 which format fits {CHATS} chats at {TOKENS:,} tokens ({need / GIB:.1f} GiB of cache)")
for name, block_bytes, weights in FORMATS:
left = BUDGET - weights_bytes(bits_per_weight(block_bytes, weights))
verdict = f"yes, {(left - need) / GIB:.2f} GiB to spare" if left >= need else "no"
print(f" {name:<5} leaves {left / GIB:>5.2f} GiB {verdict}")
print("Step 6 one big value in a row of 4,096 made-up weights")
rng = random.Random(7)
row = [rng.gauss(0, 0.02) for _ in range(4096)]
with_big = list(row)
with_big[1000] = 0.6 # 30 times the usual size, put in on purpose
print(" row bits one scale for relative error rounded to zero")
for label, data in (("no big value", row), ("one big value", with_big)):
for bits in (8, 4):
for scale_for, block_size in (("the whole row", len(data)), ("32 weights", 32)):
restored = quantize(data, bits, block_size)
zeros = sum(1 for v in restored if v == 0.0)
print(f" {label:<13} {bits:>5} {scale_for:<17} {relative_error(data, restored):>14.3f} {zeros / len(data):>14.1%}")
uv run --no-project --python 3.12 python weight_formats.pyStep 1 the problem from the KV cache guide
32 chats at 8,192 tokens need 14.0 GiB of cache
bf16 weights leave 7.43 GiB, so 16 chats fit
Step 2 bits per weight from the block layout
bf16 2 bytes per block of 1 = 16.0 bits per weight
Q8_0 34 bytes per block of 32 = 8.5 bits per weight
Q4_0 18 bytes per block of 32 = 4.5 bits per weight
Step 3 weights, space left and chats on one 24 GiB GPU
format bits/weight weights GiB left GiB chats@8,192 chats@4,096
bf16 16.00 14.17 7.43 16 33
Q8_0 8.50 7.53 14.07 32 64
Q4_0 4.50 3.99 17.61 40 80
Step 4 the mistake: count 4.0 bits and forget the scale
weights 3.54 GiB, left 18.06 GiB, chats@8,192: 41, right is 40
Step 5 which format fits 32 chats at 8,192 tokens (14.0 GiB of cache)
bf16 leaves 7.43 GiB no
Q8_0 leaves 14.07 GiB yes, 0.07 GiB to spare
Q4_0 leaves 17.61 GiB yes, 3.61 GiB to spare
Step 6 one big value in a row of 4,096 made-up weights
row bits one scale for relative error rounded to zero
no big value 8 the whole row 0.008 1.2%
no big value 8 32 weights 0.005 0.7%
no big value 4 the whole row 0.142 19.4%
no big value 4 32 weights 0.095 13.1%
one big value 8 the whole row 0.062 9.6%
one big value 8 32 weights 0.007 0.8%
one big value 4 the whole row 0.862 96.8%
one big value 4 32 weights 0.117 13.5%Where you will see this
The block layouts in step 2 are in llama.cpp's ggml-common.h. The Transformers quantization overview says an 8B model needs about 16GB for its weights in bf16, about 8GB at 8 bits and about 4 to 5GB at 4 bits. That fits step 3: 8 billion weights at 4.5 bits are 4.5 billion bytes. The same page warns that most GGUF models are dequantized to full precision as they load in Transformers, so they use as much memory as an unquantized model.
The LLM.int8() paper describes very large features in real models and one method that works around them. The vLLM page on the quantized KV cache says the cache has its own data type setting and that an FP8 cache can reduce its footprint. In our formula the cache stays at 2 bytes, so the weights are the only part we changed.
What this does not do
- This is a formula and a simulation, not a measurement. We did not load a model or use a GPU. The 24 GiB card, the 90 percent usable share and the 32 chats are our choices.
- Q8_0 and Q4_0 are llama.cpp's formats, and we only read its ggml-common.h. Another format can store its scales in another way, so 4.5 bits is not a rule for every 4-bit format.
- We say nothing about the quality or the speed of a quantized model. The next guide estimates speed from memory bandwidth. The Transformers overview says fewer bits cost some accuracy. How much depends on the model and the method, and we measured neither. The only error numbers here come from made-up weights.
- The row in step 6 is made up, and the big value is put in on purpose. The rounding is a simplified round to the nearest step with the scale kept in full precision. llama.cpp stores its scale in 2 bytes and its code is not this code, so these numbers are not what llama.cpp would give.
- The LLM.int8() paper is about large features in real models, not about the weights in our row. We use it only to say that real models can have a few very large values.
- The cache is counted at 2 bytes for each value, as in the KV cache guide, and this guide changes only the weights. The vLLM page says the cache has its own data type setting and that an FP8 cache can reduce its footprint, so an engine can store it another way and change the chat counts.
- The 90 percent usable share is a guess at what an engine keeps for other things. 32 chats at 8,192 tokens need 14.0 GiB and Q8_0 leaves 14.07 GiB, a margin of 0.07 GiB. A small change in other memory use flips the answer, so treat 32 chats at 8-bit as a close call and not as a promise.
- A real file can be bigger or smaller than the formula, and an engine may load it in another way. Check the size of the file you plan to load and the memory your engine really uses.
- The numbers belong to this one model. Another model has another parameter count and cache size.
- Model files, docs and code change. We read every source on the date below.
Memory tells you what fits. A test of your own cases tells you whether the quantized model is good enough. To build that test, see how to turn a failed agent run into an eval that blocks the release, and how to tell whether a change worked or was luck. Both use an agent as the example, and the method is the same for any model: save your cases, grade the result and count enough runs. To go back a step, read the KV cache guide. To see how fast each chat will be on the same card, read how fast each chat decodes. More guides are on the guides page.
Explore this topic in the workspace
Topic Breakdown gives you a structured revision guide for this topic and lets you highlight any phrase to open side-by-side chats with worked examples. Sign in to open it.
Open breakdownOr type LLM quantization: weight formats and GPU memory into the topic box on the home page. To see a lesson first, open the sample breakdown on Machine Learning.
Sources
All sources were checked on 15 November 2026. Read each page for the full context.
- Qwen2.5-7B-Instruct: model card
7.61B parameters, 28 layers, and 28 attention heads for Q and 4 for KV. We use the parameter count here and the head counts through the KV cache guide. Only the first part of the page was read.
- llama.cpp: ggml-common.h (commit 85c5522)
block_q8_0 is a 2 byte scale (ggml_half) and 32 int8 values. block_q4_0 is a 2 byte scale and 16 bytes for 32 values of 4 bits, and QK4_0 and QK8_0 are 32. The link is pinned to a commit, so the file can differ on the main branch.
- Transformers docs: Quantization overview
Models are usually loaded in bf16, which takes 2 bytes per parameter. An 8B model needs about 16GB for its weights in bf16, about 8GB at 8 bits and about 4 to 5GB at 4 bits, and fewer bits cost some accuracy. Most GGUF models are dequantized to full precision as they load, so they use as much memory as an unquantized model.
- LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (arXiv 2208.07339)
Only the abstract was read. It describes highly systematic emergent features in transformer language models, separate normalization constants for each inner product, and outlier feature dimensions kept in a 16-bit matrix multiplication while more than 99.9% of values are multiplied in 8 bits.
- vLLM docs: Quantized KV Cache (stable)
Quantizing the KV cache to FP8 can significantly reduce its memory footprint. The setting kv_cache_dtype takes values such as auto, which uses the model's default data type, and fp8. We did not run vLLM.
- Our guide: Will 32 chats fit on one GPU?
56 KiB of cache per token, 448 MiB for a chat of 8,192 tokens, 7.43 GiB left on the same card with 16 bit weights, and 16 chats at 8,192 tokens.