Your LLM labels each ticket for 2.2 cents. Can a small model take over? Shadow-run it in Python first
A support team has an LLM that puts every ticket into one of 6 labels, and each call costs about 2.2 cents. A public post by Amazon engineers reports that a small model took over a ticket labeling job from an LLM at 98 percent less cost. The authors say that running both models on real tickets and judging every disagreement told them more than any offline test. You want to know if a swap would hold up on your tickets, without a risky switch. By the end of this guide you can:
- Price a ticket on the LLM and on a small model, and find the yearly volume where the swap covers its fixed cost.
- Run both models in a shadow run, judge every disagreement, and see which kind of ticket the small model gets wrong.
- Pick a confidence threshold on one week of tickets, check it on a second week, and write the ship rule and the rollback.
You need Python 3.12 or newer. No GPU, no API key, no network and no packages to install.
The tickets, the stand-in for the LLM and every accuracy figure are made up. The prices come from the Amazon post. We ran no real model. This is a method on toy data, not a measurement of real systems.
Before you start
You should know Python functions, lists, dicts and f-strings, and what a classifier and a label are. If not, open these chapters first. The first three open a roadmap for that topic, and you sign in to create it. The last two are guides here.
- Text classification with naive Bayes. Count how often each word shows up under each label, then pick the label with the biggest score. Our small model is about 20 lines of it.
- Probability calibration for classifiers. A model that says 90 percent should be right about 90 percent of the time. Step 6 checks that.
- Fixed and variable costs in model serving. An instance that is always on costs the same at any volume, so the cost for each ticket falls as volume rises.
- Did the fix work, or was it luck? Compare two versions of an LLM agent. How an exact test and its p-value tell a real gap from luck. It compares two separate groups of runs. Step 3 uses a version for two models on the same tickets.
- Turn a failed agent run into an eval that blocks the release. How to turn the ship rule of step 7 into a check that blocks a release.
The question
A made-up support team uses an LLM to label each ticket as billing, outage, access, security, feature or howto. A small classifier could do the job for less. Can it take over? Do not switch and hope. Run it beside the LLM first, count where they differ, and let the evidence set the rules.
Five words we use
- Shadow run
- A new model runs on live requests next to the current one, but the customer gets only the current model's answer, and both answers are logged. The AWS docs on shadow tests describe it as routing a copy of the requests to the new variant while only the production variant's responses go back to the caller.
- Disagreement
- A ticket where the two models give different labels. Agreement does not prove that either model is right, because both can be wrong in the same way. So a person judges the disagreements. In this guide a made-up truth column plays that person.
- Confidence threshold and fallback
- The small model gives a label and a confidence from 0 to 1. At or above the threshold its label is used. Below it, the ticket goes to the LLM. The LLM is the fallback.
- Calibration
- Whether the stated confidence matches how often the model is right. The scikit-learn docs say a well calibrated classifier is right on about 80 percent of the samples it gave a probability close to 0.8.
- Break-even
- The volume of tickets where the LLM calls you stop paying for equal the fixed cost of running the small model. Below it, the swap loses money.
Worked example
Price the swap, run both models in a shadow run, judge the disagreements, set a threshold, check it on a second week, then write the rule.
| Setting | LLM runs on | Small model runs on | Customer gets |
|---|---|---|---|
| BIG | Every ticket | No | The LLM's label |
| SHADOW | Every ticket | Every ticket, and the pair is logged | The LLM's label |
| SMALL | Only tickets the small model is not sure about | Every ticket | The small model's label when it is sure, else the LLM's label |
LLM right: 238Small model right: 20Neither right: 14
Price one ticket
The Strands post reports that Amazon's LLM used 6,554 input tokens and 170 output tokens for each ticket, at $3.00 and $15.00 per million tokens. That is $0.0222 a ticket, or $22,212 per million tickets. The post says the small model ran on an instance that costs $1.408 an hour and handles about 5,600 tickets an hour. We assume the instance is always on, which the post does not say, so it costs $12,334 a year and takes about 49,056,000 tickets a year at full use.
An always-on instance costs the same at any volume, so the cost per million tickets falls as volume rises: $12,334 at 1 million tickets a year, $2,467 at 5 million, $493 at 25 million and $251 at 49,056,000. If the small model answered every ticket, the LLM calls it saves would equal the instance cost at 555,289 tickets a year. Below that volume the swap loses money, however good the model is. These figures leave out the shadow period, labeling and your own time. The post says its figures leave out the Lambda function.
Run both models for a week in SHADOW mode
The function serve() reads one setting on every ticket. In SHADOW mode it runs both models, applies the LLM's label and logs the pair, so the customer sees no change. The small model is naive Bayes trained on 600 made-up tickets. Week 1 has 1,000 tickets. The two models agree on 728 of them, 72.8 percent. That looks good and tells you little, because agreement does not say who is right.
Judge every disagreement
The models differ on 272 tickets, and a person judges each one. Here the truth column plays that person. The LLM is right on 238, the small model on 20 and neither on 14. An exact test asks how likely a gap of 238 against 20 or larger is if the two models were equally good. Both models saw the same tickets, so only the 258 tickets where exactly one model is right count, and each would be a coin flip. This is an exact test like the one in the luck guide, in a version for pairs. statsmodels calls it McNemar's test with exact set to true. The answer is p = 1.55e-48, which is extremely unlikely, so the gap is not luck. Raw agreement hid a large gap in accuracy.
Then split by kind of ticket. Clear tickets use the label's own words. The LLM is right on 98.0 percent of them and the small model on 100.0 percent. Mixed tickets also use words of another label. The LLM is right on 91.2 percent and the small model on only 51.0. Alias tickets use customer words such as hacked or frozen that were never in the 600 training tickets. The small model has learned none of these words, so only the class priors and the filler words move its answer. It is right on 16.1 percent, close to the 16.7 percent that guessing among 6 labels gives. Real tickets have no kind column. You find these groups by reading the disagreements.
Two cautions. The small model beats the LLM on clear tickets only because our stand-in LLM errs at random 2 percent of the time. That says nothing about a real LLM. And 5 tickets got the same wrong label from both models, so they are not disagreements, and comparing labels alone cannot find them. Have a person judge a random sample of the agreed tickets too.
Set a confidence threshold with a fallback
In SMALL mode a ticket goes to the small model only when its confidence is at or above the threshold. The LLM answers the rest. The table in the output scores five thresholds on week 1. At 0.00 the small model answers every ticket and is right on 74.3 percent. At 0.90 it answers 62.9 percent of the tickets and is right on 99.0 percent of those. With the LLM taking the rest, the blend is right on 96.8 percent, against 96.1 percent for the LLM alone.
The cost per million tickets is the instance cost spread over 5 million tickets a year, which is our chosen volume, plus the LLM price for every ticket the small model does not answer. At 0.90 it is $10,707, about 48 percent of the $22,212 for the LLM alone. Our rule is to take the lowest threshold from this short list whose blend is within 1 point of the LLM alone, meaning not more than 1 point lower. That is 0.90. With a fallback the small model saves only on the tickets it answers, so the break-even rises from 555,289 to 882,812 tickets a year. That uses the 62.9 percent it answers on week 1, and a different share moves it. Our 5 million is above it.
Check the threshold on a second week
A threshold picked on week 1 looks best on week 1, so we score it on week 2, which has 1,000 new tickets it was not chosen on. The small model answers 64.5 percent of them, and the blend is right on 96.7 percent, against 95.5 percent for the LLM alone. That is within 1 point.
The blend is above the LLM here because the small model is right on every clear ticket in made-up data. Do not expect that on real tickets. Also, both weeks come from the same made-up process, so week 2 cannot show drift. Real traffic can drift, and in real life week 2 needs its own judged sample.
Check that the confidence is honest
Group the week 1 tickets by the confidence the small model stated, and compare it with how often it was right. In the top two bands the stated confidence is close to how often the small model was right: stated 99.7 and 99.9 percent, right on 100.0 percent. The middle bands are not close. Tickets stated at 87.3 percent on average were right 51.1 percent of the time, and tickets stated at 65.6 percent were right 50.0 percent.
The scikit-learn docs say naive Bayes is known to be a bad estimator, so its predict_proba outputs are not to be taken too seriously. That is why step 4 counts right answers instead of trusting the printed number. We show a table and not one score because the scikit-learn page on calibration says a Brier score alone is of limited use for judging calibration.
Write the rule and the rollback
Ship the small model only if all four hold. A person judged every disagreement in a sample, and a random sample of the agreed tickets. The blend on a week the threshold was not chosen on is within your margin of the LLM alone. We used 1 point, and yours is a business choice. Your yearly volume is above the break-even. And going back is one value. In the code, setting the mode to BIG gives the LLM's label on all 1,000 week 2 tickets.
The Amazon post describes the same shape. A three way setting is read on every message, and going back was a 45 second config change, as the authors report. After the switch, keep judging a weekly sample, including tickets the small model answered with high confidence, and watch for new customer words, because alias tickets are where it failed.
Run it yourself
Put the code in one file named shadow_run.py. The script was run on 18 November 2026 with Python 3.12.16 and 3.15.2. The output was the same on both. The code uses only the standard library, so there is nothing to install.
The command uses uv, which can download Python 3.12 if you do not have it. Without uv, run python shadow_run.py in Python 3.12 or newer.
shadow_run.py prints one block for each worked step. The block for step 7 only checks the rollback, because the rule itself is not code. To try it on your tickets, keep serve() and the log. Replace make_ticket with your tickets, and SmallModel with your classifier. Replace big_label with a call to your LLM: in SHADOW mode it runs on every ticket, and in SMALL mode only on the tickets below the threshold, which is where the savings come from. A person judges the logged disagreements instead of the truth column. The accuracy and confidence tables also need judged samples of agreed tickets and of a second week, because disagreements alone cannot give them.
import math
import random
from collections import Counter
# Made up for this guide: the labels, the words, the tickets and the stand-in for the big model.
LABELS = ["billing", "outage", "access", "security", "feature", "howto"]
CORE = {
"billing": "invoice refund charge payment receipt card",
"outage": "down timeout error slow crash unavailable",
"access": "login password locked invite sso permission",
"security": "breach leak phishing token suspicious vulnerability",
"feature": "export dark integration wish roadmap request",
"howto": "setup guide configure steps docs example",
}
ALIAS = { # customer words that never appear in the training tickets
"billing": "billed bill money",
"outage": "broken frozen stuck",
"access": "signin entry blocked",
"security": "hacked stolen scam",
"feature": "missing idea addition",
"howto": "tutorial manual instructions",
}
FILLER = "the a my our please team since today again could when why hello thanks urgent customer account page app yesterday still".split()
TRAIN_MIX = {"clear": 75, "mixed": 25, "alias": 0}
TRAFFIC_MIX = {"clear": 60, "mixed": 20, "alias": 20}
BIG_RIGHT = {"clear": 0.98, "mixed": 0.90, "alias": 0.95} # chosen by us, not measured on any model
# Prices and call size as the Strands post reports them for Amazon's ticket labeling (9 November 2026).
BIG = (6554 * 3.00 + 170 * 15.00) / 1e6 # dollars per ticket: input and output tokens at $3 and $15 per million
INSTANCE_YEAR = 1.408 * 24 * 365 # one small-model instance, always on (our assumption), dollars a year
MAX_TICKETS_YEAR = 5600 * 24 * 365 # the post: about 5,600 tickets an hour on one instance
VOLUME = 5_000_000 # tickets a year, chosen by us
def money(value, places=0):
return "$" + format(value, f",.{places}f")
def make_ticket(rng, name, mix):
label = rng.choice(LABELS)
kind = rng.choices(list(mix), list(mix.values()))[0]
own = CORE[label].split()
other = CORE[rng.choice([x for x in LABELS if x != label])].split()
if kind == "clear":
words = rng.choices(own, k=3)
elif kind == "mixed":
words = rng.choices(own, k=2) + rng.choices(other, k=2)
else:
words = rng.choices(ALIAS[label].split(), k=3)
words += rng.choices(FILLER, k=5)
rng.shuffle(words)
return {"id": name, "words": words, "truth": label, "kind": kind}
def big_label(ticket):
"""The stand-in for the big model: right with a chance we chose, else a wrong label."""
rng = random.Random(f"big-{ticket['id']}")
if rng.random() < BIG_RIGHT[ticket["kind"]]:
return ticket["truth"]
return rng.choice([x for x in LABELS if x != ticket["truth"]])
class SmallModel:
"""Naive Bayes: count the words of each label, multiply the odds, pick the biggest."""
def __init__(self, tickets):
self.n = len(tickets)
self.prior = Counter(t["truth"] for t in tickets)
self.counts = {x: Counter() for x in LABELS}
for t in tickets:
self.counts[t["truth"]].update(t["words"])
self.vocab = {w for t in tickets for w in t["words"]}
self.total = {x: sum(c.values()) for x, c in self.counts.items()}
def predict(self, words):
logp = {}
for x in LABELS:
score = math.log(self.prior[x] / self.n)
for w in words:
if w in self.vocab:
score += math.log((self.counts[x][w] + 1) / (self.total[x] + len(self.vocab)))
logp[x] = score
best = max(logp, key=logp.get)
return best, 1 / sum(math.exp(v - logp[best]) for v in logp.values())
def exact_p(b, c):
"""Two sided exact test. If the models were equally good, each ticket where exactly one is right would be a coin flip."""
n, k = b + c, min(b, c)
tail, ways = 0, 1
for i in range(k + 1):
tail += ways
ways = ways * (n - i) // (i + 1)
return min(1.0, 2 * tail / 2**n)
rng = random.Random(7)
train = [make_ticket(rng, f"tr{i}", TRAIN_MIX) for i in range(600)]
week1 = [make_ticket(rng, f"te{i}", TRAFFIC_MIX) for i in range(1000)]
week2 = [make_ticket(rng, f"ho{i}", TRAFFIC_MIX) for i in range(1000)]
model = SmallModel(train)
SETTINGS = {"mode": "SHADOW", "threshold": 0.90} # read on every ticket, so going back is one edit
def serve(ticket, log):
"""The label the customer gets. BIG: big model only. SHADOW: both run, the big label is applied
and the pair is logged. SMALL: the small model answers when it is sure, else the big model."""
mode = SETTINGS["mode"]
if mode == "BIG":
return big_label(ticket)
small, conf = model.predict(ticket["words"])
if mode == "SHADOW":
big = big_label(ticket)
log.append({**ticket, "big": big, "small": small, "conf": conf})
return big
return small if conf >= SETTINGS["threshold"] else big_label(ticket)
print("Step 1 what a ticket costs")
print(f" big model: {money(BIG, 4)} a ticket, {money(BIG * 1e6)} per million")
print(f" one small-model instance: {money(INSTANCE_YEAR)} a year, about {MAX_TICKETS_YEAR:,} tickets a year at full use")
print(" tickets a year small model per million")
for tickets in (1_000_000, 5_000_000, 25_000_000, MAX_TICKETS_YEAR):
print(f" {tickets:>14,} {money(INSTANCE_YEAR / tickets * 1e6):>23}")
print(f" if the small model took every ticket, it pays for itself at {INSTANCE_YEAR / BIG:,.0f} tickets a year")
log = []
for t in week1:
serve(t, log)
n = len(log)
agree = sum(r["big"] == r["small"] for r in log)
disputed = [r for r in log if r["big"] != r["small"]]
big_won = sum(r["big"] == r["truth"] for r in disputed)
small_won = sum(r["small"] == r["truth"] for r in disputed)
print(f"Step 2 shadow run, week 1: {n} tickets, the big label is applied")
print(f" the two models agree on {agree} ({agree / n:.1%})")
print(f"Step 3 a person judges the {len(disputed)} disagreements (here the truth column plays that person)")
print(f" big right {big_won}, small right {small_won}, neither {len(disputed) - big_won - small_won}")
print(f" exact test, {big_won} against {small_won}: p = {exact_p(big_won, small_won):.2e}")
print(f" both wrong with the same label, so no disagreement: {sum(r['big'] == r['small'] != r['truth'] for r in log)}")
print(" kind of ticket tickets big right small right agree")
for kind in TRAFFIC_MIX:
rows = [r for r in log if r["kind"] == kind]
cells = [sum(r["big"] == r["truth"] for r in rows), sum(r["small"] == r["truth"] for r in rows), sum(r["big"] == r["small"] for r in rows)]
print(f" {kind:<14} {len(rows):>8} " + " ".join(f"{c / len(rows):>{w}.1%}" for c, w in zip(cells, (9, 11, 5))))
print("Step 4 small answers at or above a confidence, the big model answers the rest (week 1)")
print(" threshold small answers small right blended right cost per million")
big_acc = sum(r["big"] == r["truth"] for r in log) / n
chosen = None
for threshold in (0.0, 0.5, 0.7, 0.9, 0.99):
taken = [r for r in log if r["conf"] >= threshold]
share = len(taken) / n
small_acc = sum(r["small"] == r["truth"] for r in taken) / len(taken)
blended = sum((r["small"] if r["conf"] >= threshold else r["big"]) == r["truth"] for r in log) / n
cost = INSTANCE_YEAR / VOLUME * 1e6 + (1 - share) * BIG * 1e6
print(f" {threshold:>9.2f} {share:>13.1%} {small_acc:>11.1%} {blended:>13.1%} {money(cost):>16}")
if chosen is None and blended >= big_acc - 0.01:
chosen, chosen_share = threshold, share
print(f" big model alone: {big_acc:.1%} right, {money(BIG * 1e6)} per million. Costs assume {VOLUME:,} tickets a year.")
break_even = INSTANCE_YEAR / (BIG * chosen_share)
print(f" lowest threshold within 1 point of the big model: {chosen:.2f}, break-even {break_even:,.0f} tickets a year")
SETTINGS.update(mode="SMALL", threshold=chosen)
applied = [serve(t, None) for t in week2]
share2 = sum(model.predict(t["words"])[1] >= chosen for t in week2) / len(week2)
blended2 = sum(a == t["truth"] for a, t in zip(applied, week2)) / len(week2)
big2 = sum(big_label(t) == t["truth"] for t in week2) / len(week2)
print(f"Step 5 check {chosen:.2f} on week 2, tickets it was not chosen on")
print(f" small answers {share2:.1%}, blended right {blended2:.1%}, big model alone {big2:.1%}")
print(f" within 1 point of the big model: {'yes' if blended2 >= big2 - 0.01 else 'no'}")
print("Step 6 is the stated confidence honest? week 1, small model")
print(" stated confidence tickets stated right")
for lo, hi in ((0.0, 0.5), (0.5, 0.8), (0.8, 0.95), (0.95, 0.999), (0.999, 1.01)):
rows = [r for r in log if lo <= r["conf"] < hi]
stated = sum(r["conf"] for r in rows) / len(rows)
right = sum(r["small"] == r["truth"] for r in rows) / len(rows)
close = "]" if hi > 1 else ")"
print(f" [{lo:.3f}, {min(hi, 1):.3f}{close} {len(rows):>13} {stated:>6.1%} {right:>5.1%}")
SETTINGS["mode"] = "BIG"
print("Step 7 roll back by editing one value")
print(f" mode BIG gives the big label on all {len(week2)} week 2 tickets: {all(serve(t, None) == big_label(t) for t in week2)}")uv run --no-project --python 3.12 python shadow_run.pyStep 1 what a ticket costs
big model: $0.0222 a ticket, $22,212 per million
one small-model instance: $12,334 a year, about 49,056,000 tickets a year at full use
tickets a year small model per million
1,000,000 $12,334
5,000,000 $2,467
25,000,000 $493
49,056,000 $251
if the small model took every ticket, it pays for itself at 555,289 tickets a year
Step 2 shadow run, week 1: 1000 tickets, the big label is applied
the two models agree on 728 (72.8%)
Step 3 a person judges the 272 disagreements (here the truth column plays that person)
big right 238, small right 20, neither 14
exact test, 238 against 20: p = 1.55e-48
both wrong with the same label, so no disagreement: 5
kind of ticket tickets big right small right agree
clear 613 98.0% 100.0% 98.0%
mixed 194 91.2% 51.0% 49.5%
alias 193 94.8% 16.1% 16.1%
Step 4 small answers at or above a confidence, the big model answers the rest (week 1)
threshold small answers small right blended right cost per million
0.00 100.0% 74.3% 74.3% $2,467
0.50 80.7% 88.2% 89.5% $6,754
0.70 71.4% 93.4% 93.6% $8,819
0.90 62.9% 99.0% 96.8% $10,707
0.99 61.1% 100.0% 97.3% $11,107
big model alone: 96.1% right, $22,212 per million. Costs assume 5,000,000 tickets a year.
lowest threshold within 1 point of the big model: 0.90, break-even 882,812 tickets a year
Step 5 check 0.90 on week 2, tickets it was not chosen on
small answers 64.5%, blended right 96.7%, big model alone 95.5%
within 1 point of the big model: yes
Step 6 is the stated confidence honest? week 1, small model
stated confidence tickets stated right
[0.000, 0.500) 193 27.7% 16.1%
[0.500, 0.800) 144 65.6% 50.0%
[0.800, 0.950) 47 87.3% 51.1%
[0.950, 0.999) 272 99.7% 100.0%
[0.999, 1.000] 344 99.9% 100.0%
Step 7 roll back by editing one value
mode BIG gives the big label on all 1000 week 2 tickets: TrueWhere you will see this
The Strands post on Amazon's ticket labeling describes a similar three way setting (BEDROCK, DUAL and DECIDER) and the same advice to judge every disagreement. In its DECIDER setting the small model labels every ticket and the LLM is not called, while our SMALL setting falls back to the LLM below the threshold. Its small model is a decision model, which is not naive Bayes. For a test on real data, read the Big Iron guide. It uses 880 real documents and needs Ollama, numpy and scikit-learn, which this guide avoids. The Wavect article gives a break-even calculation with hypothetical numbers and the advice to audit a sample of the answers you accept automatically.
What this does not do
- The tickets, the words, the stand-in LLM and every accuracy figure are made up. The stand-in LLM has a 98, 90 and 95 percent chance of being right on three kinds of ticket, and we chose those numbers. We ran no real model, so we say nothing about real accuracy, real tickets or which model wins.
- The small model is naive Bayes on word counts. It is not the decision model in the Amazon post, and we did not reproduce the Amazon figures. They are the authors' report in a post dated 9 November 2026.
- The token counts, prices and 5,600 tickets an hour come from one post on one date. The always-on instance is our assumption, because the post says its endpoint autoscales. Check current prices on the Amazon Bedrock and SageMaker pricing pages. Our costs leave out the shadow week, when both models run, and also labeling, storage, other compute and your own time.
- The truth column is always right. Real people disagree, so have two people judge a sample and settle their differences.
- The exact test treats tickets as independent. Real tickets come in clusters, such as one incident that brings 50 reports. Clusters break that assumption, so do not trust a p-value this small without an analysis that allows for them.
- The five thresholds and the within 1 point rule are our choices. Both weeks come from the same made-up process, so we could not show drift.
- Sources and prices change. We read every source on the date below.
To compare two groups of runs, see the luck guide. To run your ship rule as a check, see the release gate guide. More guides are on the guides page.
Explore this topic in the workspace
Topic Breakdown gives you a structured revision guide for this topic and lets you highlight any phrase to open side-by-side chats with worked examples. Sign in to open it.
Open breakdownOr type Production ML: shadow deployment, calibration and cost per request into the topic box on the home page. To see a lesson first, open the sample breakdown on Machine Learning.
Sources
All sources were checked on 18 November 2026. Read each page for the full context.
- Strands Agents: Amazon saves millions by replacing Sonnet 4.6 with Strands Decider (9 November 2026)
6,554 input and 170 output tokens for each ticket for the LLM, $3.00 and $15.00 per million tokens, $1.408 an instance hour, about 640 ms a request and about 5,600 tickets an hour on one instance, a three way setting (BEDROCK, DUAL, DECIDER) with a 45 second rollback, and the lesson that it worked for a closed choice at high volume. It says its figures leave out the Lambda. All numbers are the authors' report.
- scikit-learn: Naive Bayes
Naive Bayes is known to be a bad estimator, so the probability outputs from predict_proba are not to be taken too seriously.
- scikit-learn: Probability calibration
A well calibrated classifier is right on about 80 percent of the samples with a predicted probability close to 0.8, and a Brier score alone is of limited use for assessing calibration.
- scikit-learn: Tuning the decision threshold for class prediction
It says never to use the same data for training the classifier and tuning the decision threshold, because of the risk of overfitting.
- statsmodels: mcnemar
With exact set to true, the McNemar test uses the binomial distribution. Our exact test counts the tickets where exactly one model is right.
- Amazon SageMaker AI: Shadow tests
A copy of the requests goes to the shadow variant, only the production variant's responses go back to the caller, and the shadow responses can be discarded or logged for offline comparison.
- Big Iron: Replacing an LLM Call With a Tiny Classifier (2 October 2026)
A test on real data: 880 labeled documents in 5 categories, embeddings from Ollama with scikit-learn logistic regression, a local 7B chat model as the baseline, a calibration section and a cascade. The author's results, which we did not rerun.
- Wavect: Small model fine-tuning, Airbnb's lesson for production classifiers (9 November 2026)
A break-even calculation with hypothetical inputs, and the advice to keep an audited sample of automatically accepted decisions and to measure the fixed decision rule on examples that did not select it.
- Amazon Bedrock pricing
Where to check current LLM prices. The numbers in this guide come from the Strands post and not from this page.
- Amazon SageMaker AI pricing
Where to check current instance prices. The numbers in this guide come from the Strands post and not from this page.
- Our guide: Did the fix work, or was it luck?
An exact test on counts and why a small gap can be luck.
- Our guide: Turn a failed agent run into an eval that blocks the release
How to run a rule as a check in CI.