All guides

Did the fix work, or was it luck? Compare two versions of an LLM agent

Your agent passed 7 of 10 runs. After a fix it passed 9 of 10. You cannot tell yet if the fix worked. By the end of this guide you can:

  • See how often luck alone makes one version of an agent look better than another.
  • Run an exact test on two pass counts and read the p-value.
  • Plan how many runs a real gain needs, and write a ship rule that can say inconclusive.

You need Python 3.12 or newer. No API key and no network. SciPy is optional, for one cross-check.

Before you start

This is episode 3. You should know Python functions, loops and the random module. If not, open these first. Each link opens a breakdown for that topic with a revision guide and side chats. Sign in to open it.

  1. Debug an LLM agent: find the first bad step in its trace. Episode 1. We reuse its refund agent.
  2. Turn a failed agent run into an eval that blocks the release. Episode 2. An eval case that grades one run. Here we count passes over many runs.
  3. Repeated trials and the binomial idea. Ten runs are a sample of a pass rate, not the pass rate.
  4. P-values for comparing two rates. Step 3 reads a p-value.

The question

In episodes 1 and 2 the refund agent guessed an amount when its order lookup was blocked, and you fixed it. Now suppose the agent is a real model, so a run can pass or fail by chance. Before the wrapper fix it passed 7 of 10 runs. After the fix it passed 9 of 10. Did the fix work?

The agent below is a simulation with a pass rate we set, so you can see what luck alone does. It is not a live model. The 7 of 10 and 9 of 10 counts are a made-up scenario, not a result from a real product, and there is no customer data.

One run does not settle this, and neither do ten. The tau-bench paper reports that agents it tested, such as gpt-4o in 2024, were inconsistent across repeated trials. So you count passes over many runs, and you ask how much of the gap luck could explain.

Four words we use

Pass rate
The share of runs that pass. The true pass rate is what you would see over a huge number of runs. Ten runs only estimate it.
Luck alone
Both versions have the same true pass rate, and any gap between the counts is chance.
p-value
If luck alone were at work, how often would the counts be at least this uneven? A small p-value means luck is a poor explanation. It is not the chance that your fix works.
Power
The chance that a test calls a real gain a win. We aim for 0.8 here. That is our choice, not a rule.

Worked example

Decide whether 9 of 10 beats 7 of 10, then plan the runs that would settle it.

  1. Read the two counts

    7 of 10 is 70 percent. 9 of 10 is 90 percent. It looks like a win, but ten runs is a small sample of the true pass rate.

  2. Ask what luck alone does

    Simulate two identical agents that both pass 70 percent of runs. Run each one 10 times, and repeat for 20,000 pairs. Count the pairs where the second agent passes 2 or more runs more than the first. That happens in 0.2307 of the pairs, about 23 in 100. A gap like ours is common when nothing changed.

  3. Run the exact test on the real counts

    fisher_two_sided(7, 10, 9, 10) returns 0.582. A p-value this large means the counts cannot separate the fix from luck. SciPy gives the same number. A p-value above 0.05 does not show that the fix did nothing. It shows that these counts cannot tell. It is not the same number as step 2, because the test does not know the true pass rate and counts a gap in either direction.

  4. Keep the rates, add runs

    70 of 100 against 90 of 100 returns p = 0.00065. The pass rates are the same as in step 3, but the runs are ten times as many, so the evidence is clear. In the same kind of simulation, identical agents show a gap of 20 or more passes in 100 runs in 0.0018 of the pairs, about 2 in 1,000.

  5. Plan the runs before you test

    For a true gain from 70 to 90 percent, the chance that the test calls it a win is 0.057 at 10 runs per version, 0.242 at 20, 0.372 at 30, 0.534 at 40, 0.640 at 50, 0.730 at 60 and 0.813 at 70. With 10 runs per version you would miss a real 20 point gain about 94 times in 100. Counting in steps of 10, 70 runs per version is the first count that reaches 0.8. You never know the true rates, so choose the smallest gain you care about and plan for it.

  6. Write the ship rule

    Fix the run count first, for example 70 per version. Ship the change only when p is under 0.05 at that count. Otherwise report inconclusive, not fixed. Keep episode 2's test for the known bug (the 403 page) as a separate check. It shows that the bug is gone. The repeated runs measure the pass rate.

Run it yourself

Put these two files in one folder. Every command here was run on 10 November 2026 with Python 3.12.16 and 3.15.2. The output was the same on both. The SciPy check ran on Python 3.12.16 with SciPy 1.18.1. Steps 2 and 4 use seed 1, so a different seed changes the last digits.

The commands use uv. Without uv, run python compare.py in a Python 3.12 or newer environment.

fix_or_luck.py holds the exact test, the luck simulation and the run planner. It uses only the standard library.

fix_or_luck.py
import math
import random


def fisher_two_sided(a_pass, a_runs, b_pass, b_runs):
    """Exact p-value. If both versions had the same pass rate, how often would luck
    alone split the passes at least as unevenly as you saw?"""
    passes, runs = a_pass + b_pass, a_runs + b_runs
    ways = math.comb(runs, passes)

    def chance(k):  # chance that exactly k of the passes land in version A
        return math.comb(a_runs, k) * math.comb(b_runs, passes - k) / ways

    seen = chance(a_pass)
    possible = range(max(0, passes - b_runs), min(a_runs, passes) + 1)
    # The 1e-7 slack keeps ties from being dropped by float rounding.
    return sum(p for p in map(chance, possible) if p <= seen * (1 + 1e-7))


def count_passes(rng, rate, runs):
    """Run a simulated agent with a known pass rate and count its passes."""
    return sum(rng.random() < rate for _ in range(runs))


def luck_rate(rate, runs, gap, trials, seed):
    """Share of identical agent pairs where the second passes gap more runs than the first."""
    rng = random.Random(seed)
    hits = 0
    for _ in range(trials):
        first = count_passes(rng, rate, runs)
        second = count_passes(rng, rate, runs)
        hits += second - first >= gap
    return hits / trials


def chance_of(passes, runs, rate):
    return math.comb(runs, passes) * rate**passes * (1 - rate) ** (runs - passes)


def power(old_rate, new_rate, runs, alpha=0.05):
    """Chance that a test with this many runs per version calls a real gain a win (p under alpha)."""
    return sum(
        chance_of(old, runs, old_rate) * chance_of(new, runs, new_rate)
        for old in range(runs + 1)
        for new in range(old + 1, runs + 1)
        if fisher_two_sided(old, runs, new, runs) < alpha
    )

compare.py runs the six steps. Each printed line starts with its step number.

compare.py
from fix_or_luck import fisher_two_sided, luck_rate, power

print("Step 1  before:", 7 / 10, " after:", 9 / 10)
print("Step 2  identical agents, 2 or more extra passes in 10 runs:", luck_rate(0.7, 10, 2, 20_000, seed=1))
print("Step 3  p-value, 7 of 10 against 9 of 10:", round(fisher_two_sided(7, 10, 9, 10), 3))
print("Step 4  p-value, 70 of 100 against 90 of 100:", round(fisher_two_sided(70, 100, 90, 100), 5))
print("Step 4  identical agents, 20 or more extra passes in 100 runs:", luck_rate(0.7, 100, 20, 20_000, seed=1))

print("Step 5  runs per version and the chance to detect a true 70 to 90 percent gain")
runs = 10
while (chance := power(0.7, 0.9, runs)) < 0.8:
    print(f"        {runs:>3}  {chance:.3f}")
    runs += 10
print(f"        {runs:>3}  {chance:.3f}  first count at 0.8 or more")
Terminal
uv run --no-project --python 3.12 python compare.py
Output
Step 1  before: 0.7  after: 0.9
Step 2  identical agents, 2 or more extra passes in 10 runs: 0.2307
Step 3  p-value, 7 of 10 against 9 of 10: 0.582
Step 4  p-value, 70 of 100 against 90 of 100: 0.00065
Step 4  identical agents, 20 or more extra passes in 100 runs: 0.0018
Step 5  runs per version and the chance to detect a true 70 to 90 percent gain
         10  0.057
         20  0.242
         30  0.372
         40  0.534
         50  0.640
         60  0.730
         70  0.813  first count at 0.8 or more

As a cross-check, SciPy's fisher_exact gives the same two p-values.

Terminal
uv run --no-project --python 3.12 --with scipy python -c "from scipy.stats import fisher_exact; print(round(fisher_exact([[7, 3], [9, 1]]).pvalue, 3), round(fisher_exact([[70, 30], [90, 10]]).pvalue, 5))"
Output
0.582 0.00065

How the exact test works

Start from the 16 passes in the 20 runs. If both versions were equally good, those 16 passes could land in either version by chance. Python's math.comb counts the ways to choose which runs hold them. Each split gets a chance, and the p-value adds up the chance of every split that is no more likely than the one you saw. That is the two-sided rule that SciPy documents for this test.

The planner uses the same test. For every pair of results, it asks if the test would call it a win, and it adds up how likely that pair is when the new version really passes 90 percent and the old one 70 percent. For a fuller treatment of testing and planning evals, see Adding Error Bars to Evals.

What this does not do

  • The agent here is a simulation with a pass rate we set. It is not a live model, and the counts 7 of 10 and 9 of 10 are a scenario, not a result from a real product.
  • The test assumes the runs are independent and that both versions get the same mix of tasks. Shared state, a warm cache or a changed task set breaks that.
  • If you look after every run and stop as soon as p dips under 0.05, you give luck many chances to look like a win. Fix the run count first.
  • If the model behind your agent changes between the two sets of runs, you are also testing that change. Run both versions close in time.
  • Our run count is for Fisher's exact test. SciPy lists Barnard's and Boschloo's exact tests as more powerful alternatives for 2 by 2 tables, so they may need fewer runs.
  • Pass or fail hides what happened on the way and what each run cost. Episode 1's trace checker looks at the steps. Nothing here looks at cost.

Next: retry a tool call safely

When a run fails because a tool call timed out, the harness may try again, and a retry can repeat a change the first call already made. Episode 4 adds an idempotency key to a refund call and checks what the shop recorded. Read Retry a timed-out agent tool call without refunding twice.

The same question comes up for a retriever. A fix that moves 10 test questions from 5 to 10 right proves little when you built the fix from those 10. See that case in Your RAG bot gave a wrong answer. Was it retrieval or generation? Find out in Python.

The same question comes up when you swap in a cheaper model. Here the two versions see the same tickets, so the counts that matter are the tickets where exactly one version is right. See that case in Your LLM labels each ticket for 2.2 cents. Can a small model take over? Shadow-run it in Python first.

Explore this topic in the workspace

Topic Breakdown gives you a structured revision guide for this topic and lets you highlight any phrase to open side-by-side chats with worked examples. Sign in to open it.

Open breakdown

Or type Evaluating LLM agents with repeated runs into the topic box on the home page. To see a lesson first, open the sample breakdown on Machine Learning. Missed the earlier episodes? Read how to find the first bad step in a trace and how to turn a failed run into a release gate.

Sources

All sources were checked on 10 November 2026. Read each page for the full context.

  • Python docs: math.comb

    comb(n, k) is the number of ways to choose k items from n items without repetition and without order.

  • SciPy docs: scipy.stats.fisher_exact

    Fisher's exact test on a 2 by 2 table. The p-value is the probability, under the null hypothesis, of a table at least as extreme as the one observed. The two-sided p-value adds the probability of every table that is no more likely than the observed one. Barnard's and Boschloo's exact tests are listed as more powerful alternatives.

  • arXiv: Adding Error Bars to Evals

    Abstract only. Evaluations are experiments. The paper gives formulas for analyzing evaluation data, measuring differences between two models and planning an evaluation experiment, and recommends reporting results so that statistical noise is minimized.

  • arXiv: tau-bench

    Abstract only. It proposes pass^k to measure the reliability of agent behavior over multiple trials. It reports that gpt-4o function calling agents succeeded on under 50 percent of tasks and were inconsistent (pass^8 under 25 percent in retail). The paper is from 2024, so those models are old.