One slow LLM call is blocking your queue. Measure it and fix it in Python
Some chats get their replies late, and the model provider is not down. If your queue consumer calls an LLM, one slow call can make every message behind it wait. By the end of this guide you can:
- Measure how long fast messages wait behind slow LLM calls, with a median and a 95th percentile.
- Explain why more workers help only up to a point when the messages of one chat must stay in order.
- Write a rule for how many workers to run and what to alert on.
You need Python 3.12 or newer. No API key, no network, no message broker and no packages to install.
Before you start
You should know Python functions, lists and dictionaries, what a message queue and a consumer are, and that an LLM API call can take seconds. If not, open these chapters first. Each link opens a breakdown for that topic with a revision guide and side chats. Sign in to open it.
- Message queues and consumer groups. A consumer takes messages from a queue and does the work.
- Queueing basics: arrival rate, service time and utilization. How busy a worker is decides how long the line gets.
- Percentiles: why the median hides the tail. We read the median and the 95th percentile of the wait.
- How LLM API latency varies. One call can take 2 seconds and the next can take 45.
- Retry a timed-out agent tool call without refunding twice. Optional, episode 4. A message that is delivered twice is the same repeat problem, seen from the queue side.
The question
A chat service reads messages from one queue, in order. Each message needs one LLM call. Most calls take 2 seconds, and now and then one takes 45 seconds. Users whose messages were quick still see late replies. Is the cause the provider, the number of workers or the queue design?
The queue below is a simulation. We chose the call times, the share of slow calls and the arrival rate. It is not a real broker, a live model or customer data, and the times are not measurements of any provider.
Four words we use
- Queue wait
- The time from a message arriving to the start of its LLM call. We measure it for the fast messages.
- Median and 95th percentile
- The median is the middle wait: half the waits are shorter. The 95th percentile (p95) is a wait that about 95 out of 100 waits are at or below. It shows the slow tail that the median hides. Our code takes the value at index int(q × n) of the sorted list. Other definitions differ a little.
- Worker
- One LLM call in progress at a time. Four workers means up to four calls at once.
- Head-of-line blocking
- A queue held up by the item at its front. Wikipedia defines it for network packets. Our queue has the same shape, but it is not a network.
Worked example
Replay the same 100 messages, change one thing at a time, and read the wait of the fast messages.
Set up the queue
The scenario is 100 chat messages from 10 chats, one every 8 seconds. 87 of the calls take 2 seconds and 13 take 45 seconds. The mean call is 7.59 seconds. Little's law says the long-term average number of items in a system is the average arrival rate times the average time each one spends in it. Applied to the consumer, that is 7.59 seconds divided by 8 seconds, about 0.95 calls in progress at a time. One consumer is busy 95 percent of the time, so it has almost no spare time. Part of the wait in step 2 is plain overload, not only a slow call at the front. Steps 4 and 7 show what is left when the load is shared or lowered.
Run one consumer
The consumer takes messages in order. Look only at the 87 fast messages. Their wait is the time from arriving to the start of their call. The median is 55 seconds, the 95th percentile is 105 seconds and the longest is 126 seconds. A call that takes 2 seconds waited up to 126 seconds. Each 45 second call holds the consumer while the next messages pile up, and with the consumer busy 95 percent of the time the line takes a long time to clear.
Add workers with no order rule
Four workers take four messages at once. The wait of the fast messages is 0 at the median, the 95th percentile and the longest. The price is order. In 6 pairs of messages from one chat, the later message finished first. In the pair the code prints, message 20 in chat 5 (a 45 second call) finished at 205 seconds, and the later message 24 (a 2 second call) finished at 194 seconds. The user would get the reply to the later message first. A chat cannot allow that.
Keep each chat in order
Add one rule: a message waits while another message of its chat is running. Four workers now give a median wait of 0, a 95th percentile of 13 seconds and a longest wait of 50 seconds, with 0 pairs out of order. The wait is not 0, because a fast message that arrives while a slow call of its own chat is running must wait for it. That wait is the price of keeping order.
Add more workers
With the rule on, the 95th percentile of the wait is 105 seconds for 1 worker, 29 for 2, 13 for 3, 13 for 4 and 13 for 6. It stops falling at 3 workers. With 4 and with 6 workers, all 5 fast messages that still waited had a slow call from their own chat in the way. With 3 workers, 5 of the 7 did. A worker cannot take a message the rule holds back, so more workers cannot remove that wait.
Check that it is not one lucky seed
A seed makes the random messages repeat. Step 6 runs the same test with 200 seeds. For one consumer, the median of the 95th percentile waits is 81 seconds, and the values run from 37 to 261 seconds. For four workers with the rule, the median is 0 seconds, and the values run from 0 to 29 seconds. The seed in steps 1 to 5 is a little worse than the middle run for one consumer. Treat the exact numbers as belonging to this scenario, and treat the gap between the two cases as the lesson.
Lower the load
Now messages arrive every 12, 20 or 30 seconds instead of every 8. With one consumer, the median wait is 3, 0 and 0 seconds, and the 95th percentile is 69, 25 and 15 seconds. The median looks fine and the tail does not. At 30 seconds apart, one consumer is busy about 25 percent of the time, and a fast message that arrives during a 45 second call still waits up to 30 seconds. Four workers with the rule give a 95th percentile of 0 at all three spacings, and the longest waits are 33, 25 and 15 seconds.
Write the rule
First, set the worker count from the busy share: the mean call time divided by the time between messages and by the number of workers. Keep it well under 100 percent. Here 4 workers give 7.59 divided by 8 × 4, about 24 percent. We chose 4 because step 5 stops improving at 3 workers and 4 leaves some room. Pick your own number and check it against the 95th percentile. Second, if order matters, allow one call per chat, or per key, at a time. Third, alert on the 95th percentile of the time from a message arriving to its call starting, not on the median and not on the mean call time. In step 2 the mean call was 7.59 seconds, the median wait was 55 seconds and the 95th percentile was 105 seconds.
Run it yourself
Put the code in one file named queue_sim.py. Every command here was run on 13 November 2026 with Python 3.12.16 and 3.15.2. The output was the same on both. The code uses only the standard library, so there is nothing to install.
The command uses uv. Without uv, run python queue_sim.py in Python 3.12 or newer.
queue_sim.py replays the queue and prints one block for each of steps 1 to 7. The out of order column counts pairs of messages from one chat where the later message finished first.
import heapq
import random
import statistics
FAST, SLOW = 2.0, 45.0 # seconds one LLM call takes in this scenario
def make_messages(n=100, chats=10, gap=8.0, slow_share=0.1, seed=7):
"""One message arrives every gap seconds. Each one belongs to a chat and needs one LLM call."""
rng = random.Random(seed)
return [
{"id": i, "chat": rng.randrange(chats), "arrive": i * gap, "work": SLOW if rng.random() < slow_share else FAST}
for i in range(n)
]
def run(messages, workers, one_per_chat):
"""Replay the queue. A free worker takes the oldest waiting message it is allowed to take.
With one_per_chat, a message waits while another message of its chat is running."""
coming = sorted(({**m} for m in messages), key=lambda m: m["arrive"]) # copies, so a run leaves messages alone
waiting, running, busy_chats, done = [], [], set(), []
while coming or waiting or running:
now = min(
coming[0]["arrive"] if coming else float("inf"),
running[0][0] if running else float("inf"),
)
while coming and coming[0]["arrive"] <= now:
waiting.append(coming.pop(0))
while running and running[0][0] <= now:
_, _, msg = heapq.heappop(running)
msg["finish"] = now
busy_chats.discard(msg["chat"])
done.append(msg)
for msg in list(waiting):
if len(running) >= workers:
break
if one_per_chat and msg["chat"] in busy_chats:
continue
waiting.remove(msg)
msg["start"] = now
busy_chats.add(msg["chat"])
heapq.heappush(running, (now + msg["work"], msg["id"], msg))
return done
def pct(values, q):
"""The value at index int(q * n) of the sorted list. Other percentile definitions differ a little."""
values = sorted(values)
return values[min(len(values) - 1, int(q * len(values)))]
def fast_waits(done):
"""Seconds each fast message waited before its call started."""
return [m["start"] - m["arrive"] for m in done if m["work"] == FAST]
def late_pairs(done):
"""Pairs of messages from one chat where the later message finished first."""
by_chat = {}
for m in sorted(done, key=lambda m: m["id"]):
by_chat.setdefault(m["chat"], []).append(m)
return [
(a, b)
for msgs in by_chat.values()
for i, a in enumerate(msgs)
for b in msgs[i + 1:]
if b["finish"] < a["finish"]
]
def behind_own_chat(done):
"""Of the fast messages that waited, how many arrived while a slow call from their own chat was unfinished."""
slow = [m for m in done if m["work"] == SLOW]
waited = [m for m in done if m["work"] == FAST and m["start"] > m["arrive"]]
blocked = [m for m in waited if any(s["chat"] == m["chat"] and s["arrive"] < m["arrive"] < s["finish"] for s in slow)]
return len(waited), len(blocked)
def plural(n, word):
return f"{n} {word}" + ("s" if n > 1 else "")
def show(label, done):
waits = fast_waits(done)
print(f" {label:<30}{pct(waits, 0.5):>5.0f}{pct(waits, 0.95):>6.0f}{max(waits):>6.0f}{len(late_pairs(done)):>14}")
HEADER = f" {'':<30}{'p50':>5}{'p95':>6}{'max':>6}{'out of order':>14}"
GAP = 8.0
messages = make_messages(gap=GAP)
slow_count = sum(m["work"] == SLOW for m in messages)
mean_call = statistics.mean(m["work"] for m in messages)
print("Step 1 the setup")
print(f" {len(messages)} messages from {len({m['chat'] for m in messages})} chats, one every {GAP:.0f} s")
print(f" {len(messages) - slow_count} fast calls ({FAST:.0f} s) and {slow_count} slow calls ({SLOW:.0f} s)")
print(f" mean call {mean_call:.2f} s, so one consumer is busy {mean_call / GAP:.0%} of the time")
print("Step 2 one consumer (wait of the fast messages, in seconds)")
print(HEADER)
show("one consumer", run(messages, 1, True))
print("Step 3 four workers, no order rule")
print(HEADER)
loose = run(messages, 4, False)
show("4 workers, no rule", loose)
early, later = late_pairs(loose)[0]
print(f" chat {early['chat']}: message {early['id']} ({early['work']:.0f} s call) finished at {early['finish']:.0f} s,")
print(f" but message {later['id']} ({later['work']:.0f} s call) finished first, at {later['finish']:.0f} s")
print("Step 4 four workers, one call per chat at a time")
print(HEADER)
show("4 workers, one per chat", run(messages, 4, True))
print("Step 5 more workers, one call per chat (p95 wait of fast messages)")
for workers in (1, 2, 3, 4, 6):
done = run(messages, workers, True)
waited, blocked = behind_own_chat(done)
print(
f" {plural(workers, 'worker')}: p95 {pct(fast_waits(done), 0.95):>3.0f} s, "
f"{waited:>2} fast messages waited, {blocked} of them behind a slow call of their own chat"
)
print("Step 6 200 different random seeds (p95 wait of fast messages)")
for workers in (1, 4):
p95s = [pct(fast_waits(run(make_messages(gap=GAP, seed=seed), workers, True)), 0.95) for seed in range(200)]
print(f" {plural(workers, 'worker')}: median {statistics.median(p95s):.0f} s, lowest {min(p95s):.0f} s, highest {max(p95s):.0f} s")
print("Step 7 messages arrive less often (wait of the fast messages, in seconds)")
print(f" {'':<30}{'p50':>5}{'p95':>6}{'max':>6}")
for gap in (8, 12, 20, 30):
spaced = make_messages(gap=gap)
for workers in (1, 4):
waits = fast_waits(run(spaced, workers, True))
label = f"every {gap} s, {plural(workers, 'worker')}"
print(f" {label:<30}{pct(waits, 0.5):>5.0f}{pct(waits, 0.95):>6.0f}{max(waits):>6.0f}")
uv run --no-project --python 3.12 python queue_sim.pyStep 1 the setup
100 messages from 10 chats, one every 8 s
87 fast calls (2 s) and 13 slow calls (45 s)
mean call 7.59 s, so one consumer is busy 95% of the time
Step 2 one consumer (wait of the fast messages, in seconds)
p50 p95 max out of order
one consumer 55 105 126 0
Step 3 four workers, no order rule
p50 p95 max out of order
4 workers, no rule 0 0 0 6
chat 5: message 20 (45 s call) finished at 205 s,
but message 24 (2 s call) finished first, at 194 s
Step 4 four workers, one call per chat at a time
p50 p95 max out of order
4 workers, one per chat 0 13 50 0
Step 5 more workers, one call per chat (p95 wait of fast messages)
1 worker: p95 105 s, 81 fast messages waited, 9 of them behind a slow call of their own chat
2 workers: p95 29 s, 16 fast messages waited, 5 of them behind a slow call of their own chat
3 workers: p95 13 s, 7 fast messages waited, 5 of them behind a slow call of their own chat
4 workers: p95 13 s, 5 fast messages waited, 5 of them behind a slow call of their own chat
6 workers: p95 13 s, 5 fast messages waited, 5 of them behind a slow call of their own chat
Step 6 200 different random seeds (p95 wait of fast messages)
1 worker: median 81 s, lowest 37 s, highest 261 s
4 workers: median 0 s, lowest 0 s, highest 29 s
Step 7 messages arrive less often (wait of the fast messages, in seconds)
p50 p95 max
every 8 s, 1 worker 55 105 126
every 8 s, 4 workers 0 13 50
every 12 s, 1 worker 3 69 89
every 12 s, 4 workers 0 0 33
every 20 s, 1 worker 0 25 57
every 20 s, 4 workers 0 0 25
every 30 s, 1 worker 0 15 30
every 30 s, 4 workers 0 0 15Where you will see this
In Kafka, the design docs say a topic is divided into totally ordered partitions, each consumed by exactly one consumer within each subscribing consumer group at any given time. If that consumer works through messages one by one, a slow message holds up the rest of its partition, as in step 2. This is the shape of head-of-line blocking in networks, though not the same system.
If your code makes the LLM call inside the poll loop, the call time counts toward the time between poll() calls. The consumer configs say that if poll() is not called before max.poll.interval.ms expires, the consumer is considered failed and the group rebalances. The page lists a default of 300000 milliseconds, which is 5 minutes.
Celery's docs say that for many long tasks you want a prefetch multiplier of one, and that for a mix of long and short tasks the best option is two separately configured worker nodes, with tasks routed by run time. We did not test routing. It needs you to know which messages will be slow before you run them.
The Kafka 4.2 notes say share groups are production ready. Consumers in a share group consume records without each partition going to one consumer. The notes say to use them when records are processed one at a time, rather than as part of an ordered stream. The Kafka 4.0 javadoc says share groups share records more flexibly at the expense of record ordering, and that a record is held under a lock of 30 seconds by default. So a share group gives up the per-chat order from step 4. A 45 second call is also longer than that default, and when the lock runs out the record can go to another consumer, so the call may run twice. That page is for version 4.0, so check the default for your version. We did not run a share group.
What this does not do
- The queue is a simulation. We chose the call times (2 and 45 seconds), the share of slow calls and the arrival rate. It is not a real broker, a live model or customer data, and nothing here measures any provider, model or queue product.
- Every call takes one of two fixed times, and messages arrive evenly spaced. Real call times are a spread and real traffic comes in bursts. This model does not show either.
- Order is kept per chat only. A queue that needs one global order cannot use the rule in step 4.
- A provider rate limit (a 429 error) caps your total calls and can make retries pile up. This model does not show that.
- A real consumer must also handle crashes and messages delivered twice. Episode 4 covers the retry side.
- We did not run asyncio code, a real broker or a Kafka share group, so nothing here claims results for them.
- The numbers belong to these inputs. Change FAST, SLOW, the slow share and the gap in the code, run it again, and check your own load before you pick a worker count.
Next, see how to retry a timed-out agent tool call without refunding twice, which covers a message that is delivered twice. If each step of an agent run is a queue message, see how to find the first bad step in its trace. If you serve the model yourself, the number of chats open at once is also limited by GPU memory. See how many chats fit on one GPU. A long prompt can also stall the streams of other chats on one GPU. See why the first token is slow.
Explore this topic in the workspace
Topic Breakdown gives you a structured revision guide for this topic and lets you highlight any phrase to open side-by-side chats with worked examples. Sign in to open it.
Open breakdownOr type Queue workers and concurrency for LLM calls into the topic box on the home page. To see a lesson first, open the sample breakdown on Machine Learning. More guides are on the guides page.
Sources
All sources were checked on 13 November 2026. Read each page for the full context.
- Apache Kafka 4.4 docs: Design
A topic is divided into totally ordered partitions, each of which is consumed by exactly one consumer within each subscribing consumer group at any given time.
- Apache Kafka 4.4 docs: Consumer configs
max.poll.interval.ms is the maximum delay between invocations of poll() when using consumer group management. If poll() is not called before it expires, the consumer is considered failed and the group rebalances. The page lists a default of 300000.
- Apache Kafka 4.2 upgrade notes
Queues for Kafka (KIP-932) is production-ready in Apache Kafka 4.2. Share groups let consumers consume records without each partition going to one consumer, with per-record acknowledgement. The notes say to use them when records are processed one at a time, rather than as part of an ordered stream. The page is for an older release, so read the notes for yours.
- Apache Kafka 4.0 javadoc: KafkaShareConsumer
Share groups give more flexible sharing of records than a consumer group, at the expense of record ordering. A record is held under a time-limited lock, 30 seconds by default. The page also calls the feature early access, which was true for 4.0 only.
- Celery 5.7 docs: Optimizing
For many long tasks, use a prefetch multiplier of one. For a mix of long and short tasks, the best option is two separately configured worker nodes, with tasks routed by run time. The docs also say to monitor queue lengths and set alerts.
- Little's law (Wikipedia)
The long-term average number of customers in a stationary system equals the long-term average arrival rate times the average time a customer spends in it, L = λW. This is a general statement. We use it only to read the busy share in step 1. It does not prove the simulation.
- Head-of-line blocking (Wikipedia)
A networking term for a performance-limiting effect that occurs when a queue of packets is held up by the first packet in the queue. Our queue has the same shape but is not a network.
- Python docs: random
Python uses the Mersenne Twister as its core generator. It is completely deterministic, which is why a seed repeats the same messages. The docs say the module's generators should not be used for security purposes.