All guides

Will speculative decoding speed up your LLM chat? Size drafts, acceptance and load in Python

The tokens per second guide ended at a ceiling set by the bytes each step reads. Speculative decoding tries to get past it. A small draft model guesses several tokens, and the big model checks all of them in one step. The Leviathan et al. paper reports streams 2 to 3 times faster for one model. A real setup can match that, fall short of it, or get slower. By the end of this guide you can:

  • Size the speedup from three numbers, the acceptance rate, the draft length and the cost of a draft step, and pick a draft length.
  • Explain why the same setting gets slower as more chats share the GPU, and why a length tuned on one chat can be slower than no drafting at 64 chats.
  • Read the acceptance counters your engine reports, and check the formula against a paper and a measured table before you turn drafting on.

You need Python 3.12 or newer. No GPU, no API key, no network and no packages to install.

Before you start

You should know Python functions, loops and f-strings, and the cache size, decode ceiling and compute limit from the guides we link and do not repeat. If not, open these chapters first. Each link opens a breakdown for that topic with a revision guide and side chats. Sign in to open it.

  1. How an LLM generates text one token at a time. A reply is one pass over the prompt, then one pass for each new token.
  2. Compute-bound and memory-bound work in GPUs. A step is limited by its arithmetic or by its memory reads, whichever is slower.
  3. Batching and scheduling in LLM serving. Chats share one step, so one chat's extra tokens add to every chat's wait.
  4. Expected value and independent events. The tokens a round gives is an average over rounds that each stop at the first miss.
  5. Will 32 chats fit on one GPU? Size the KV cache in Python. Where the 56 KiB of cache per token comes from.
  6. You fit 32 chats on one GPU. How fast will each one decode? Estimate it in Python. The decode ceiling that drafting tries to get past, and the card we reuse.
  7. A long prompt makes your chat wait before the first word. Why? Size prefill and decode in Python. The guide before this one. It shows a step turning compute bound above about 100 tokens.

The question

A team serves a 7 billion parameter chat model on one 24 GiB GPU. Someone proposes a 0.5 billion parameter draft model from the same family. Will streams get faster? How many tokens should the draft model guess in each round? Does the answer change when 64 chats share the card instead of one? And your engine reports an acceptance rate. What does that number mean?

The first answers are a formula on a card we chose. The last checks use the tables of Leviathan et al. and a public test on an RTX 3090 that someone else measured. We ran no GPU.

Five words we use

Draft model
A small model that guesses the next tokens cheaply. The big model is the one whose answers you want. Engines also draft with n-gram matching and extra prediction layers, and the llama.cpp docs list several. We use a small model.
Round
One cycle. The draft model guesses a few tokens one after another. The big model then checks all of them in one step. It accepts the guesses in order until the first miss and adds one token of its own. So a round gives at least 1 token and at most one more than the number of guesses.
Draft length
The number of tokens guessed in a round. vLLM calls it num_speculative_tokens, and llama.cpp has --spec-draft-n-max. The code calls it gamma, as the paper does.
Acceptance rate
In the paper, alpha is the chance that the big model accepts one guessed token, taken to be the same for every token. Engines report other numbers. Step 6 shows how they differ.
Cost ratio
The paper calls it c. It is the time of one draft step divided by the time of one big step. A small c means drafting is cheap.

Worked example

Count the tokens in a round, price a round on a card, add chats, pick the draft length for the load, then check the formula against a paper, an engine report and a measured table.

The same 7 drafts at acceptance rate 0.8, with more chats on the card

1 chat: 2.86 times as fast as no drafting. The check step is memory bound.

4 chats: 2.84 times as fast as no drafting. The check step is memory bound.

16 chats: 2.42 times as fast as no drafting. The check step is compute bound.

32 chats: 1.46 times as fast as no drafting. The check step is compute bound.

64 chats: 0.87 times as fast as no drafting. The check step is compute bound.

The length of no drafting. A bar inside the box is slower than that.The speed with drafting. A bar past the box is faster.

  1. Count the tokens one round gives

    A round has two parts. The draft model guesses gamma tokens one after another. Then the big model checks all of them in one step. It accepts guesses in order until the first miss, and adds one token of its own. If each guess is accepted with the same chance alpha, given that the guesses before it were accepted, a round gives (1 - alpha^(gamma + 1)) / (1 - alpha) tokens on average. This is equation 1 of Leviathan et al. The code also plays 200,000 rounds with a fixed random seed and counts the tokens. The two stay within 0.003 of each other at 4 drafts: 1.938 and 1.941 at alpha 0.5, and 3.362 and 3.359 at alpha 0.8. So at alpha 0.8, 4 drafts give about 3.4 tokens for each big step. That is not a speedup yet, because the drafts cost time too.

  2. Price a round on the chosen card

    We reuse the card, the big model and the 56 KiB of cache per token of the earlier serving guides: 1,000 GB per second and 100 TFLOPS (both our choice) and Qwen2.5-7B-Instruct. The draft model is Qwen2.5-0.5B-Instruct. Its model card gives 0.49 billion parameters. Its config.json gives 24 layers, 2 key-value heads, a hidden size of 896 and 14 attention heads, so each head has 64 values. A token then needs 2 × 24 × 2 × 64 × 2 bytes, which is 12 KiB of cache. With 1,024 tokens in the cache of one chat, a step of the big model takes 15.28 ms and a step of the draft model takes 0.99 ms. Each is the slower of its arithmetic and its memory reads, as in the first token guide. Their ratio is the cost ratio c = 0.065. A round with gamma drafts costs gamma draft steps plus one big step that checks gamma + 1 tokens. While the card is memory bound, that big step costs about the same as a plain step, because it reads the same weights once. The speedup is the tokens per round times the plain step time, divided by the round time. Read the table. At alpha 0.8, 1 draft gives 1.69 times, 4 give 2.67, 7 give 2.86 and 8 give 2.85. More drafts stop paying because each extra draft costs a draft step and is accepted less often. At alpha 0.3, 7 drafts are slower than no drafting, 0.98 times. With 1 draft the speedup is (1 + alpha) / (1 + c), so any alpha above c = 0.065 helps. That holds only while the big step costs no more than a plain step, which is true while it is memory bound. The next step shows when it stops.

  3. Add chats

    Keep the 7 drafts that were best for one chat at alpha 0.8, and let more chats share the card, each with 1,024 tokens in its cache. The big step must now check 8 tokens for every chat. At 4 chats that is 32 tokens, the card is still memory bound and the speedup is 2.84 times. At 16 chats it is 128 tokens. The arithmetic takes 19.5 ms and the memory reads take 16.2 ms, so the step turns compute bound and the speedup falls to 2.42. At 32 chats it is 1.46. At 64 chats it is 0.87 times, slower than no drafting. A plain step for 64 chats does only 64 tokens of arithmetic and takes 19.0 ms, set by its memory reads. The check step does 512 tokens of arithmetic, which takes 77.9 ms, far past the memory time, and some of that work is for guesses that get rejected. The weights and caches of both models take 19.3 GiB at 64 chats, so this fits a 24 GiB card, before the memory that activations and the engine need. This is the compute limit of the first token guide seen from the other side. Drafting spends spare arithmetic, and the spare runs out as chats are added. The vLLM docs say the feature is for medium-to-low QPS, memory-bound workloads, and their method table rates a draft model as high gain at low QPS and medium gain at high QPS.

  4. Pick the draft length for the load

    Search each load for the draft length with the best speedup. At alpha 0.8 the best length falls from 7 drafts for one chat to 5 for 16 chats and 1 for 64 chats, and the speedup falls from 2.86 to 2.70 to 1.61. At alpha 0.4 the best lengths are 2, 2 and 1 (1.38, 1.36 and 1.25 times). At alpha 0.6 they are 4, 3 and 1 (1.83, 1.78 and 1.43). In this formula, leaving drafting off never wins, because a single draft costs little. But a length set on one chat and left alone is slower than no drafting at 64 chats (0.87 times in step 3), while a length of 1 gives 1.61 times. So the best draft length depends on the load. The vLLM docs list a method called dynamic speculative decoding for traffic that moves up and down. We read only its row in the method table and did not test it.

  5. Check the formula against the paper

    The formula comes from Leviathan et al. (2023). Table 1 of the paper gives the speed and the extra arithmetic for six pairs of alpha and drafts, with a free draft model. The code reproduces all six rows. At alpha 0.9 and 10 drafts it gives 6.86 times the speed and 1.60 times the operations, as the paper does. Table 4 gives 12 runs of T5 models on a TPU v4, one chat at a time, with alpha, drafts and c from those runs. The code gets the paper's formula column to within 0.07. The paper prints alpha and c to two decimals. The measured speed was 0.68 to 1.29 times the formula. Three of the 12 runs were below it, all on translation: 0.68 and 0.72 with T5-large and 0.86 with T5-base. The highest was 1.29, T5-small on summarization. The paper says the differences come from how well its implementation and the baseline were optimized, and from the assumption that alpha is the same for every token, which it calls an approximation. So read the formula as a guide and not as a promise. The paper's draft models had c from 0.02 to 0.11, and ours is 0.065.

  6. Read the acceptance numbers your engine reports

    vLLM does not report alpha. The vLLM docs describe per-request counters, still marked experimental, and server totals at /metrics: vllm:spec_decode_num_drafts_total, vllm:spec_decode_num_draft_tokens_total and vllm:spec_decode_num_accepted_tokens_total. The per-request fields are the mean acceptance length (1 plus the accepted drafts per step), the draft acceptance rate (accepted drafts divided by proposed drafts) and a histogram of how many drafts each step accepted. The docs show an example response with 3 drafts per step: 43 steps, 129 proposed drafts, 10 accepted, and a histogram of 39, 1, 0 and 3 steps. It is an example in the docs, not our measurement. The code turns it into three numbers. The mean is 1.233 tokens per step, the draft acceptance rate is 0.078, and the alpha that gives 1.233 tokens per step in the formula is 0.19. Reading the 0.078 as alpha would predict 1.084 tokens per step and not 1.233, because the share also counts every guess after a miss, and none of those can be accepted. Then look at the histogram. A constant alpha of 0.19 would give 34.8, 6.6, 1.3 and 0.3 steps. The example has 39, 1, 0 and 3. In 42 of the 43 steps the drafts were all wrong or all right, so a constant alpha describes the average and not the shape. Look at the histogram before you pick a draft length from an average.

  7. Check it against someone else's measurements

    A blogger tested llama.cpp on one RTX 3090 with Qwen3.5 9B and 27B models in 4-bit files, using MTP drafts, which are prediction layers trained with the model. The post gives tokens per second without and with drafting on two builds, one from before llama.cpp 1.0 (build 12199) and the 1.0 release (build 12359). The code divides them. On the 1.0 build the 27B model went 1.70 and 1.79 times faster. On the older build the same 27B setup gained only 1.11 and 1.17 times, and the 9B model got slower, 0.79 and 0.91 times. The post says the model accepted about 32 to 33 percent of the guesses with the 27B model and 27 to 28 percent with the 9B model on both builds, so the post reads the change as faster checking of several tokens at once and not as better guesses. It points to a new kernel for steps of 2 to 24 tokens on RTX 30-series and newer cards. The post does not say how it counted acceptance, and step 6 shows why the share and alpha differ. One trial with a 0.8B draft model for the 27B model made it slower, from about 40 to 30.7 tokens per second (0.77 times), and fewer than half of the guesses were accepted. Our formula cannot say any of this, because it prices every step by its arithmetic and its memory reads alone. A real engine has other costs, such as the speed of the kernel that checks a few tokens at once. These are one person's runs on one card, and we did not repeat them. Measure it on your own engine before you turn drafting on.

Run it yourself

Put the code in one file named speculative.py. Every command here was run on 21 November 2026 with Python 3.12.16 and 3.15.2. The output was the same on both. The code uses only the standard library, so there is nothing to install, and it runs in about a second.

The command uses uv. Without uv, run python speculative.py in Python 3.12 or newer.

speculative.py holds its numbers inside the file and prints one block for each of steps 1 to 7. To try your own card, change BANDWIDTH and FLOPS. To size other models, change the parameter counts and the cache bytes for each token. To try another load, change CONTEXT and the lists of chats and alpha values.

speculative.py
import random

# Big model: Qwen2.5-7B-Instruct, with the numbers of the earlier serving guides.
TARGET_PARAMS = 7.61e9  # model card
TARGET_KV = 2 * 28 * 4 * 128 * 2  # 56 KiB per token, from the KV cache guide
# Draft model: Qwen2.5-0.5B-Instruct. Model card: 0.49B parameters. config.json: 24 layers, 2 KV heads, 896 / 14 = 64 per head.
DRAFT_PARAMS = 0.49e9
DRAFT_KV = 2 * 24 * 2 * 64 * 2  # 12 KiB per token

# Chosen for this example. They are not measurements.
BANDWIDTH = 1000e9  # a card that reads 1,000 GB per second
FLOPS = 100e12  # a card that does 100 trillion bf16 operations per second
CONTEXT = 1024  # tokens already in each chat's cache
GIB = 2**30


def tokens_per_round(alpha, gamma):
    """Expected tokens one round gives: the accepted drafts plus the one token the big model adds."""
    if alpha == 1:
        return gamma + 1
    return (1 - alpha ** (gamma + 1)) / (1 - alpha)


def simulate(alpha, gamma, rounds, seed=7):
    """Draft gamma tokens, accept them in order with chance alpha each, stop at the first miss."""
    rng = random.Random(seed)
    total = 0
    for _ in range(rounds):
        accepted = 0
        while accepted < gamma and rng.random() < alpha:
            accepted += 1
        total += accepted + 1
    return total / rounds


def draft_seconds(chats):
    """One draft step for every chat. The slower of its arithmetic and its memory reads."""
    reads = DRAFT_PARAMS * 2 + chats * CONTEXT * DRAFT_KV
    return max(2 * DRAFT_PARAMS * chats / FLOPS, reads / BANDWIDTH)


def target_seconds(tokens_each, chats):
    """One step of the big model with tokens_each new tokens for every chat."""
    reads = TARGET_PARAMS * 2 + chats * CONTEXT * TARGET_KV
    return max(2 * TARGET_PARAMS * chats * tokens_each / FLOPS, reads / BANDWIDTH)


def speedup(alpha, gamma, chats):
    """Time for one token without drafting, divided by the time per token with it. 0 drafts means off."""
    if gamma == 0:
        return 1.0
    round_seconds = gamma * draft_seconds(chats) + target_seconds(gamma + 1, chats)
    return tokens_per_round(alpha, gamma) * target_seconds(1, chats) / round_seconds


print("Step 1  tokens per round: the formula against a simulation (4 drafts, 200,000 rounds each)")
print("        alpha  formula  simulated")
for alpha in (0.5, 0.7, 0.8, 0.9):
    print(f"        {alpha:>5.1f} {tokens_per_round(alpha, 4):>8.3f} {simulate(alpha, 4, 200_000):>10.3f}")

c = draft_seconds(1) / target_seconds(1, 1)
print(f"Step 2  one chat, {CONTEXT:,} tokens in cache: a step takes {target_seconds(1, 1) * 1000:.2f} ms, a draft step {draft_seconds(1) * 1000:.2f} ms, so c = {c:.3f}")
print(f"        with 1 draft the speedup is (1 + alpha) / (1 + c), so it is above 1 only when alpha is above c = {c:.3f}")
print("        speedup by alpha (columns) and drafts per round (rows)")
alphas = (0.3, 0.5, 0.6, 0.7, 0.8, 0.9)
print("        drafts " + "".join(f"{a:>7.1f}" for a in alphas))
for gamma in range(1, 9):
    print(f"        {gamma:>6} " + "".join(f"{speedup(a, gamma, 1):>7.2f}" for a in alphas))
best_one = max(range(1, 9), key=lambda gamma: speedup(0.8, gamma, 1))
print(f"        best for alpha 0.8: {best_one} drafts, {speedup(0.8, best_one, 1):.2f} times")

print(f"Step 3  keep those {best_one} drafts at alpha 0.8 and add chats ({CONTEXT:,} tokens in each cache)")
print("        chats  memory GiB  tokens checked  arithmetic ms  memory ms  bound by  plain step ms  speedup")
for chats in (1, 4, 16, 32, 64):
    tokens = chats * (best_one + 1)
    compute, memory = 2 * TARGET_PARAMS * tokens / FLOPS, (TARGET_PARAMS * 2 + chats * CONTEXT * TARGET_KV) / BANDWIDTH
    bound = "compute" if compute > memory else "memory"
    used = (TARGET_PARAMS * 2 + DRAFT_PARAMS * 2 + chats * CONTEXT * (TARGET_KV + DRAFT_KV)) / GIB
    print(f"        {chats:>5} {used:>11.1f} {tokens:>15} {compute * 1000:>14.1f} {memory * 1000:>10.1f}  {bound:<8} {target_seconds(1, chats) * 1000:>13.1f} {speedup(0.8, best_one, chats):>8.2f}")

print("Step 4  the best number of drafts for each load. 0 drafts means leave drafting off")
print("        alpha  chats  best drafts  speedup")
for alpha in (0.4, 0.6, 0.8):
    for chats in (1, 16, 64):
        best = max(range(9), key=lambda gamma: speedup(alpha, gamma, chats))
        print(f"        {alpha:>5.1f} {chats:>6} {best:>12} {speedup(alpha, best, chats):>8.2f}")

# Leviathan et al. 2023, Table 1: alpha, drafts, operations factor, speed factor (draft cost set to 0).
TABLE_1 = [(0.6, 2, 1.53, 1.96), (0.7, 3, 1.58, 2.53), (0.8, 2, 1.23, 2.44), (0.8, 5, 1.63, 3.69), (0.9, 2, 1.11, 2.71), (0.9, 10, 1.60, 6.86)]
# Table 4: task, draft model, temperature, drafts, alpha, c, expected speed, measured speed.
TABLE_4 = [
    ("EnDe", "T5-small", 0, 7, 0.75, 0.02, 3.2, 3.4), ("EnDe", "T5-base", 0, 7, 0.8, 0.04, 3.3, 2.8),
    ("EnDe", "T5-large", 0, 7, 0.82, 0.11, 2.5, 1.7), ("EnDe", "T5-small", 1, 7, 0.62, 0.02, 2.3, 2.6),
    ("EnDe", "T5-base", 1, 5, 0.68, 0.04, 2.4, 2.4), ("EnDe", "T5-large", 1, 3, 0.71, 0.11, 2.0, 1.4),
    ("CNNDM", "T5-small", 0, 5, 0.65, 0.02, 2.4, 3.1), ("CNNDM", "T5-base", 0, 5, 0.73, 0.04, 2.6, 3.0),
    ("CNNDM", "T5-large", 0, 3, 0.74, 0.11, 2.0, 2.2), ("CNNDM", "T5-small", 1, 5, 0.53, 0.02, 1.9, 2.3),
    ("CNNDM", "T5-base", 1, 3, 0.55, 0.04, 1.8, 2.2), ("CNNDM", "T5-large", 1, 3, 0.56, 0.11, 1.6, 1.7),
]
print("Step 5  checked against a paper (Leviathan et al., 2023)")
print("        Table 1, drafts cost nothing")
print("        alpha  drafts  speed  paper  operations  paper")
for alpha, gamma, paper_ops, paper_speed in TABLE_1:
    speed = tokens_per_round(alpha, gamma)
    print(f"        {alpha:>5.1f} {gamma:>7} {speed:>6.2f} {paper_speed:>6.2f} {(gamma + 1) / speed:>11.2f} {paper_ops:>6.2f}")
print("        Table 4, T5 models. c is the cost of a draft step against a big step")
print(f"        {'task':<6} {'draft':<8} {'temp':>4} {'drafts':>7} {'alpha':>6} {'c':>5} {'formula':>8} {'paper':>6} {'measured':>9} {'measured / formula':>19}")
ratios, gaps = [], []
for task, draft, temperature, gamma, alpha, cost, paper_formula, measured in TABLE_4:
    formula = tokens_per_round(alpha, gamma) / (1 + gamma * cost)
    ratios.append(measured / formula)
    gaps.append(abs(formula - paper_formula))
    print(f"        {task:<6} {draft:<8} {temperature:>4} {gamma:>7} {alpha:>6.2f} {cost:>5.2f} {formula:>8.2f} {paper_formula:>6.1f} {measured:>9.1f} {measured / formula:>19.2f}")
print(f"        the formula is within {max(gaps):.2f} of the paper's formula column. Measured speed was {min(ratios):.2f} to {max(ratios):.2f} times the formula")


def acceptance(steps, accepted, drafted, histogram, gamma):
    """The counters vLLM reports for one request, and the alpha that gives the same mean tokens per step."""
    length = 1 + accepted / steps
    low, high = 0.0, 0.999
    for _ in range(60):
        middle = (low + high) / 2
        low, high = (middle, high) if tokens_per_round(middle, gamma) < length else (low, middle)
    alpha = (low + high) / 2
    print(f"        mean tokens per step {length:.3f}, share of drafts accepted {accepted / drafted:.3f}, alpha {alpha:.2f}")
    print(f"        reading the share as alpha would predict {tokens_per_round(accepted / drafted, gamma):.3f} tokens per step, not {length:.3f}")
    print("        accepted per step  steps seen  steps if alpha were constant")
    for count, seen in enumerate(histogram):
        chance = alpha**count * (1 - alpha) if count < gamma else alpha**gamma
        print(f"        {count:>17} {seen:>11} {chance * steps:>29.1f}")


print("Step 6  reading the counters of one request (the example response in the vLLM docs, 3 drafts per step)")
acceptance(43, 10, 129, [39, 1, 0, 3], 3)

# Measured by someone else with llama.cpp on one RTX 3090, tokens per second: without drafting, with MTP drafts.
MEASURED = [
    ("27B, build 12199, short prompt", 40.6, 45.1),
    ("27B, build 12199, long prompt", 39.8, 46.5),
    ("27B, build 12359, short prompt", 40.4, 68.5),
    ("27B, build 12359, long prompt", 40.2, 71.9),
    ("9B,  build 12199, short prompt", 121.8, 96.5),
    ("9B,  build 12199, long prompt", 119.0, 108.1),
    ("9B,  build 12359, short prompt", 122.1, 122.3),
    ("9B,  build 12359, long prompt", 119.4, 146.8),
]
print("Step 7  measured by someone else: drafting can speed a stream up or slow it down")
print("        setting                           off     on  on / off")
for name, off, on in MEASURED:
    print(f"        {name:<30} {off:>6.1f} {on:>6.1f} {on / off:>8.2f}")
print(f"        one trial with a 0.8B draft model for the 27B: about 40 without, 30.7 with, {30.7 / 40:.2f} times")
Terminal
uv run --no-project --python 3.12 python speculative.py
Output
Step 1  tokens per round: the formula against a simulation (4 drafts, 200,000 rounds each)
        alpha  formula  simulated
          0.5    1.938      1.941
          0.7    2.773      2.777
          0.8    3.362      3.359
          0.9    4.095      4.094
Step 2  one chat, 1,024 tokens in cache: a step takes 15.28 ms, a draft step 0.99 ms, so c = 0.065
        with 1 draft the speedup is (1 + alpha) / (1 + c), so it is above 1 only when alpha is above c = 0.065
        speedup by alpha (columns) and drafts per round (rows)
        drafts     0.3    0.5    0.6    0.7    0.8    0.9
             1    1.22   1.41   1.50   1.60   1.69   1.78
             2    1.23   1.55   1.73   1.94   2.16   2.40
             3    1.19   1.57   1.82   2.12   2.47   2.88
             4    1.13   1.54   1.83   2.20   2.67   3.25
             5    1.08   1.49   1.80   2.22   2.78   3.54
             6    1.03   1.43   1.75   2.20   2.84   3.75
             7    0.98   1.37   1.69   2.16   2.86   3.91
             8    0.94   1.31   1.63   2.10   2.85   4.03
        best for alpha 0.8: 7 drafts, 2.86 times
Step 3  keep those 7 drafts at alpha 0.8 and add chats (1,024 tokens in each cache)
        chats  memory GiB  tokens checked  arithmetic ms  memory ms  bound by  plain step ms  speedup
            1        15.2               8            1.2       15.3  memory            15.3     2.86
            4        15.4              32            4.9       15.5  memory            15.5     2.84
           16        16.1             128           19.5       16.2  compute           16.2     2.42
           32        17.2             256           39.0       17.1  compute           17.1     1.46
           64        19.3             512           77.9       19.0  compute           19.0     0.87
Step 4  the best number of drafts for each load. 0 drafts means leave drafting off
        alpha  chats  best drafts  speedup
          0.4      1            2     1.38
          0.4     16            2     1.36
          0.4     64            1     1.25
          0.6      1            4     1.83
          0.6     16            3     1.78
          0.6     64            1     1.43
          0.8      1            7     2.86
          0.8     16            5     2.70
          0.8     64            1     1.61
Step 5  checked against a paper (Leviathan et al., 2023)
        Table 1, drafts cost nothing
        alpha  drafts  speed  paper  operations  paper
          0.6       2   1.96   1.96        1.53   1.53
          0.7       3   2.53   2.53        1.58   1.58
          0.8       2   2.44   2.44        1.23   1.23
          0.8       5   3.69   3.69        1.63   1.63
          0.9       2   2.71   2.71        1.11   1.11
          0.9      10   6.86   6.86        1.60   1.60
        Table 4, T5 models. c is the cost of a draft step against a big step
        task   draft    temp  drafts  alpha     c  formula  paper  measured  measured / formula
        EnDe   T5-small    0       7   0.75  0.02     3.16    3.2       3.4                1.08
        EnDe   T5-base     0       7   0.80  0.04     3.25    3.3       2.8                0.86
        EnDe   T5-large    0       7   0.82  0.11     2.50    2.5       1.7                0.68
        EnDe   T5-small    1       7   0.62  0.02     2.26    2.3       2.6                1.15
        EnDe   T5-base     1       5   0.68  0.04     2.35    2.4       2.4                1.02
        EnDe   T5-large    1       3   0.71  0.11     1.93    2.0       1.4                0.72
        CNNDM  T5-small    0       5   0.65  0.02     2.40    2.4       3.1                1.29
        CNNDM  T5-base     0       5   0.73  0.04     2.62    2.6       3.0                1.15
        CNNDM  T5-large    0       3   0.74  0.11     2.02    2.0       2.2                1.09
        CNNDM  T5-small    1       5   0.53  0.02     1.89    1.9       2.3                1.22
        CNNDM  T5-base     1       3   0.55  0.04     1.80    1.8       2.2                1.22
        CNNDM  T5-large    1       3   0.56  0.11     1.54    1.6       1.7                1.10
        the formula is within 0.07 of the paper's formula column. Measured speed was 0.68 to 1.29 times the formula
Step 6  reading the counters of one request (the example response in the vLLM docs, 3 drafts per step)
        mean tokens per step 1.233, share of drafts accepted 0.078, alpha 0.19
        reading the share as alpha would predict 1.084 tokens per step, not 1.233
        accepted per step  steps seen  steps if alpha were constant
                        0          39                          34.8
                        1           1                           6.6
                        2           0                           1.3
                        3           3                           0.3
Step 7  measured by someone else: drafting can speed a stream up or slow it down
        setting                           off     on  on / off
        27B, build 12199, short prompt   40.6   45.1     1.11
        27B, build 12199, long prompt    39.8   46.5     1.17
        27B, build 12359, short prompt   40.4   68.5     1.70
        27B, build 12359, long prompt    40.2   71.9     1.79
        9B,  build 12199, short prompt  121.8   96.5     0.79
        9B,  build 12199, long prompt   119.0  108.1     0.91
        9B,  build 12359, short prompt  122.1  122.3     1.00
        9B,  build 12359, long prompt   119.4  146.8     1.23
        one trial with a 0.8B draft model for the 27B: about 40 without, 30.7 with, 0.77 times

Where you will see this

In vLLM you turn it on with a speculative config and set num_speculative_tokens, and the acceptance metrics page explains the counters of step 6. In llama.cpp the setting is --spec-draft-n-max, and the llama.cpp speculative decoding doc lists the ways to draft. The formula and the six rows of step 5 are in the paper. The measured rows of step 7 are in the homebrewserver.club test. What this guide adds is one script with printed output that puts the load, the paper, the engine counters and a measured table in one place.

What this does not do

  • This is a formula on a chosen card, not a measurement. The 1,000 GB/s and 100 TFLOPS card, the 1,024 token cache, the alpha values and the draft lengths are our choices. We ran no GPU, no vLLM and no llama.cpp.
  • We did not check that any engine accepts Qwen2.5-0.5B-Instruct as a draft for Qwen2.5-7B-Instruct, or measure its acceptance rate, so no alpha here is a claim about that pair.
  • The formula takes alpha as the same for every token and every chat. The paper calls that an approximation, and the histogram in step 6 is not constant.
  • A draft step costs only its weight and cache reads. A real small model also pays kernel launch, scheduling, sampling and the work to keep its own cache current, and step 7 shows slowdowns that the formula cannot produce.
  • The check step costs the slower of its arithmetic and its memory reads, as in the first token guide. It leaves out attention arithmetic, reading the cache for the new tokens and the fixed cost of a step. Every chat has the same cache, alpha and draft length, and each round waits for all of them.
  • The paper shows that its speculative sampling leaves the output distribution unchanged. We did not test output quality. The vLLM docs also list a lossy verifier, and we say nothing about it.
  • We cover one small draft model. We do not cover tree drafting, n-gram drafts, prediction heads, the memory a draft model takes beyond its weights and cache, or other engines. MTP appears only as a measured case in step 7.
  • The measured rows in step 7 are one blogger's runs of llama.cpp on one RTX 3090 with Qwen3.5 files, not Qwen2.5 and not a GPU server. We did not repeat them or check whether other cards and builds agree.
  • Docs, model files and posts change. We read every source on the date below.

This is the fifth guide on serving a model. The KV cache guide sizes the cache and the quantization guide sizes the weights. The tokens per second guide sizes the stream, the first token guide sizes the wait, and this one asks whether guessing ahead lifts the stream. A long job can also stall everything behind it in a queue, see why one slow LLM call stalls your queue. More guides are on the guides page.

Explore this topic in the workspace

Topic Breakdown gives you a structured revision guide for this topic and lets you highlight any phrase to open side-by-side chats with worked examples. Sign in to open it.

Open breakdown

Or type How LLM inference works: speculative decoding, batching and the memory bound into the topic box on the home page. To see a lesson first, open the sample breakdown on Machine Learning.

Sources

All sources were checked on 21 November 2026. Read each page for the full context.