Your RAG bot gave a wrong answer. Was it retrieval or generation? Find out in Python
A help center bot told a customer they have 30 days to send back a jacket. The outdoor gear policy says 45. It is easy to open the prompt first. Before you do, find out which part failed: did the retriever never send the right passage, or did the answer go wrong even with it? By the end of this guide you can:
- Print what a retriever sent for one wrong answer, and see why the chunk with the answer was never scored.
- Sort wrong answers into retrieval misses, generation misses and questions the help center does not answer, and read hit at k and MRR.
- Test a retriever fix on questions it was not built from, and see why 10 out of 10 on the questions you tuned on proves little.
You need Python 3.12 or newer. No GPU, no API key, no network and no packages to install.
The shop, the questions and the recorded answers are made up. The bot is recorded text, not a model, and the retriever is a small keyword scorer, not an embedding search. This is a method on toy data, not a measurement of real systems.
Before you start
You should know Python functions, lists, dicts and f-strings, what a RAG app does, and what top k means. If not, open these chapters first. Each link opens a breakdown for that topic with a revision guide and side chats. Sign in to open it.
- Retrieval-augmented generation basics. The app finds chunks of text first and puts the best ones in the prompt before the question.
- Keyword search and BM25. How a keyword scorer ranks chunks, and why a chunk with no shared word gets no score.
- Evaluating retrieval: hit rate and MRR. The two numbers this guide prints for every retriever.
- Test sets for LLM apps. Where test questions come from, and why the ones you tune on are the easy ones.
- Chunking text for retrieval. How a help center is cut into the pieces that the retriever scores.
The question
A made-up outdoor shop has a help center of 14 chunks. A customer asks, "How long do I have to send back a jacket?" The bot answers, "You have 30 days from delivery to return a jacket." The outdoor gear chunk says jackets have a 45 day window. Which part failed, and what do you fix first?
Five words we use
- Chunk
- A short piece of the help center text. The retriever scores every chunk, and the best ones go into the prompt before the question.
- Gold chunk
- The chunk that a person says holds the answer to a question. We chose the gold chunk for every question ourselves. One question has none, because no chunk answers it.
- Retrieval miss and generation miss
- A retrieval miss is a question whose gold chunk is not in the top 3 chunks, so the model never saw it. A generation miss is a question whose gold chunk is in the top 3, but the answer still lacks the fact it needed.
- Hit at k and MRR
- Hit at k counts the questions whose gold chunk is in the top k. MRR, the mean reciprocal rank, is the average of 1 divided by the rank of the gold chunk, and a question whose gold chunk is not found counts as 0.
- BM25
- A keyword scorer. A chunk gets points for each word of the question that it contains. A rare word is worth more than a common one, a word that repeats counts a little less each time, and a long chunk is marked down. A chunk with no word of the question scores 0. Ours is a small teaching version, not a library.
Worked example
Print what the retriever sent, sort the wrong answers into buckets, fix the retriever one change at a time, then test the fixes on questions they were not built from.
See what the retriever sent
The question 'How long do I have to send back a jacket?' becomes the words long, send, back and jacket after we drop common words such as how and do. Only 2 of the 14 chunks share a word with it. The chunk refund-timing has 'back' in 'go back to the original payment method'. The chunk damaged-item has 'send' in 'we send a replacement'. The gold chunk, returns-outdoor, says 'Jackets' and '45 day window', so it shares no word with the question and scores 0. A keyword scorer ranks only chunks with a score, so that chunk is not in the ranking at any k.
Our recorded answer says 30 days. It is written text, not model output, so we do not say what a model would do with these 2 chunks. We can say what the retriever sent: nothing about jackets.
The score of refund-timing is 1.060, and you can check it by hand. The idf of 'back' is ln(1 + (14 - 1 + 0.5) / (1 + 0.5)) = 2.303, because 1 of the 14 chunks has it. The chunk is 14 words long after we drop common words, and the average is 14.429. So the score is 2.303 × 1 / (1 + 1.2 × (0.25 + 0.75 × 14 / 14.429)) = 1.060, and the code prints the same number.
Sort the recorded answers into buckets
We wrote 11 questions, each with its gold chunk (or none), the fact a good answer must contain, and a recorded answer. The rule has four outcomes, checked in this order. No gold chunk: the help center does not answer it. Gold chunk not in the top 3: retrieval miss. Gold chunk in the top 3 but the fact missing from the answer: generation miss. Otherwise: ok.
The count is 3 retrieval misses (the jacket, the tent guarantee and the broken boots), 1 generation miss, 1 question the help center does not answer and 6 ok. The generation miss is the Canada question. The right chunk was in the top 3, and the recorded answer left out who pays the taxes. The question the help center does not answer is the price match one, where the recorded answer makes up a policy. For that one the fix is to write the page, not to change the retriever or the prompt.
These counts come from 11 questions and answers that we wrote, so they say nothing about which failure is more common in a real system. The rule is a toy too. It looks for one phrase, so a correct answer in other words would be called a miss. A real system needs a person or a checked judge to read the answers. The papers we cite below give no single split. One says it depends on the model, and one found that generation mattered more than retrieval accuracy on its two datasets. So this guide gives a method, not a rate.
Ask for more chunks
Hit at 3 is 7 of 10 and hit at 5 is also 7 of 10, on the 10 questions that have a gold chunk. The three misses get back 2, 0 and 0 chunks, because this scorer returns only chunks that share a word with the question. When the gold chunk shares no word, no k can find it.
This holds for a keyword scorer. A retriever that always returns k chunks, such as a nearest neighbor search, can bring the gold chunk in with a bigger k. To check your own system, print the rank of the gold chunk at a large k. If it is absent, a bigger k will not help. If it is low, a bigger k or a reranker might.
Fix the retriever one change at a time
We made three changes and counted after each one. The first adds the chunk title to the chunk text. The second adds a small word list for the question: send and back to return, guarantee to warranty, broken to damaged, discount to promo, delivery to shipping, swap to exchange and money to refund. The third cuts one trailing s, so 'Jackets' matches 'jacket'.
On the 10 tuning questions, hit at 1 goes 5, 7, 9, 10 and MRR goes 0.600, 0.700, 0.900, 1.000. The gold ranks show what each change did. The title moved two answers from rank 2 to rank 1 and fixed no miss. The word list found the guarantee and broken questions. Plural stemming found the jacket question, the headline failure, because jacket now matches Jackets.
We chose these three changes after looking at the same 3 misses, so 10 of 10 is built in. It only says the retriever now fits the 10 questions it was made from.
Test on questions the changes were not built from
We wrote two more sets of 6 questions, both before we ran any change on them. The first set was written from the docs, with the chunks in front of us. At baseline all 6 gold chunks were already in the top 3, so the changes could not add much. One question moved from rank 2 to rank 1 when plural stemming was added, and MRR went from 0.917 to 1.000. None of the 8 words in our list appears in these questions, so this set did not test the list at all. Writing a question while you read the chunk reuses the chunk's words, and that makes the set too easy to show a gain.
The second set uses plain customer words, such as cash, coupon and backpack. At baseline hit at 1 is 1 of 6 and MRR is 0.444. After all three changes hit at 1 is 2 of 6 and MRR is 0.556. The title changed nothing, and the word list changed nothing because none of its 8 words is in these questions either. Plural stemming moved the tent question from rank 3 to rank 1. The question about cash in a bank account is still never found, because the right chunk shares no word with it. So changes that took the tuning questions from 5 to 10 right at rank 1 took these from 1 to 2 of 6.
These sets are small. One question moves hit at 1 by one sixth, and we wrote both sets after reading the chunks. Do not read 0.444 or 0.556 as a score for a retriever. The point is the gap between the numbers you tune on and the numbers on new questions. Get new questions from real support tickets or search logs.
Write the rule
Before you change anything, count which bucket your wrong answers fall in. If the gold chunk is missing from the top k, work on retrieval: the words, the chunks or the index. If it is there and the answer is still wrong, look at the prompt, the order of the chunks and the model. If no chunk holds the answer, write the page.
Lost in the middle is one thing to check after retrieval. The paper reports that models often do best when the relevant passage is at the start or the end of a long input. Our inputs are short and we ran no model, so this guide does not show it.
Keep the failing questions as saved test cases. Test a fix on questions from real users, not on the questions you wrote while reading the chunks, and do not call 10 out of 10 a win when you tuned on those 10.
Run it yourself
Put the code in one file named rag_triage.py. Every command here was run on 16 November 2026 with Python 3.12.16 and 3.15.2. The output was the same on both. The code uses only the standard library, so there is nothing to install. We also wrote a second scorer from the same formula with a different structure, and it ranked the chunks the same way for all 92 pairs of retriever and question.
The command uses uv. Without uv, run python rag_triage.py in Python 3.12 or newer.
rag_triage.py holds its data inside the file and prints one block for each of steps 1 to 5. Step 6 is the rule, so it has no code. To try your own cases, change CHUNKS, CASES and the two lists of questions.
"""A wrong RAG answer: retrieval miss or generation miss?
Standard library only. No network, no API key, no embedding model, no language model.
The help center, the questions and the recorded answers are made up for this guide.
The "bot" is a list of recorded example answers, not model output.
The retriever is a small teaching version of Okapi BM25 over 14 chunks, not a library.
"""
import math
import re
from collections import Counter
K1, B = 1.2, 0.75 # common BM25 defaults, as in Lucene's BM25Similarity
STOP = {"a", "an", "the", "do", "does", "i", "my", "me", "we", "you", "to", "of", "can", "is", "it", "how", "what", "when", "and", "or", "for", "on", "in", "with", "have", "will", "who", "where", "there"}
CHUNKS = [
("returns-general", "Returns", "Most items can be returned within 30 days of delivery in original condition. Start a return from your account page."),
("returns-outdoor", "Outdoor gear returns", "Jackets, tents, sleeping bags and packs have a 45 day window because testing takes time. Items must be clean and unused."),
("refund-timing", "Refund timing", "Refunds go back to the original payment method within 5 to 7 business days after we receive the item."),
("shipping-standard", "Shipping", "Standard shipping takes 3 to 5 business days. Express shipping takes 1 to 2 business days."),
("shipping-international", "International shipping", "We ship to 40 countries. Duties and taxes are paid by the buyer at delivery. Delivery takes 7 to 14 business days."),
("order-cancel", "Cancel an order", "You can cancel an order within 1 hour of placing it. After that the order is packed and cancellation is no longer possible. Use the returns page instead."),
("exchange", "Exchanges", "To swap a size or color, return the item and place a new order. We do not ship exchanges before the return arrives."),
("warranty", "Warranty", "Tents and packs carry a 2 year warranty against defects in materials and stitching. Wear from normal use is not covered."),
("gift-cards", "Gift cards", "Gift cards never expire and cannot be refunded for cash."),
("promo-codes", "Promo codes", "One promo code per order. Codes cannot be applied after an order is placed."),
("price-adjust", "Price adjustment", "If an item drops in price within 14 days of purchase, contact support for the difference."),
("damaged-item", "Damaged or wrong item", "If your item arrives damaged or wrong, tell us within 7 days with a photo. We send a replacement or refund at no cost."),
("sizing", "Sizing help", "Our jackets run a half size small. Use the size chart on each product page before you order."),
("store-credit", "Store credit", "Returns past the window can be converted to store credit at our discretion within 90 days."),
]
# A person picks the gold chunk (None: no chunk holds the answer) and the fact a good answer must
# contain. The last column is the answer the bot gave. All made up.
CASES = [
("How long do I have to send back a jacket?", "returns-outdoor", "45 day", "You have 30 days from delivery to return a jacket."),
("When will my money come back after I return something?", "refund-timing", "5 to 7 business days", "Your refund goes to the original payment method in 5 to 7 business days after we get the item."),
("Do you deliver to Canada and who pays the taxes?", "shipping-international", "buyer", "Yes, we ship to 40 countries. Delivery takes 7 to 14 business days."),
("Can I cancel the order I placed this morning?", "order-cancel", "1 hour", "Orders can be cancelled within 1 hour of placing them. After that, use the returns page."),
("Does the tent come with a guarantee?", "warranty", "2 year", "Our items are covered by a 30 day return policy."),
("My boots showed up broken, what now?", "damaged-item", "7 days", "Sorry about that. Please contact support."),
("Can I use two discount codes?", "promo-codes", "one promo code", "Only one promo code can be used per order."),
("How fast is express delivery?", "shipping-standard", "1 to 2 business days", "Express shipping takes 1 to 2 business days."),
("I bought the wrong size, can I swap it?", "exchange", "return the item and place a new order", "Return the item and place a new order for the size you want."),
("Do gift cards expire?", "gift-cards", "never expire", "Gift cards never expire."),
("Will you match a lower price from another store?", None, "", "Yes, we match any lower price within 14 days of purchase."),
]
TUNING = [(question, gold) for question, gold, _, _ in CASES if gold]
# Written after reading the chunks and before any fix was run.
FROM_THE_DOCS = [
("What is the deadline to return a sleeping bag?", "returns-outdoor"),
("Where does my refund go and how many days does it take?", "refund-timing"),
("My pack arrived damaged, how do I report it?", "damaged-item"),
("Will a jacket fit true to size?", "sizing"),
("Is the price protected if it drops after I buy?", "price-adjust"),
("Can I turn an expired return into store credit?", "store-credit"),
]
# Also written before any fix was run, in plain customer words that the chunks do not use.
CUSTOMER_WORDS = [
("What is the refund window for my tent?", "returns-outdoor"),
("The zipper on my backpack came apart after a few months. Is that covered?", "warranty"),
("How long until the cash shows up in my bank account?", "refund-timing"),
("Do you ship to Germany?", "shipping-international"),
("Can I get a different color of the same jacket?", "exchange"),
("Can I use my coupon after I already paid?", "promo-codes"),
]
SYNONYMS = {"send": "return", "back": "return", "guarantee": "warranty", "broken": "damaged", "discount": "promo", "delivery": "shipping", "swap": "exchange", "money": "refund"}
VARIANTS = [
("baseline", {}),
("+ chunk title", {"title": True}),
("+ synonyms", {"title": True, "synonyms": True}),
("+ plural stemming", {"title": True, "synonyms": True, "stem": True}),
]
def tokens(text, stem=False):
words = [w for w in re.findall(r"[a-z0-9]+", text.lower()) if w not in STOP]
return [w[:-1] if stem and len(w) > 3 and w.endswith("s") and not w.endswith("ss") else w for w in words]
def make_scorer(title=False, synonyms=False, stem=False):
"""Return scores(question): one BM25 score per chunk. A chunk with no shared word scores 0."""
docs = [Counter(tokens(f"{name}. {text}" if title else text, stem)) for _, name, text in CHUNKS]
lengths = [sum(d.values()) for d in docs]
average = sum(lengths) / len(docs)
doc_freq = Counter(word for d in docs for word in d)
def scores(question):
query = tokens(question, stem)
if synonyms:
query += [SYNONYMS[w] for w in query if w in SYNONYMS]
result = []
for d, length in zip(docs, lengths):
score = 0.0
for word in set(query):
if word in d:
idf = math.log(1 + (len(docs) - doc_freq[word] + 0.5) / (doc_freq[word] + 0.5))
score += idf * d[word] / (d[word] + K1 * (1 - B + B * length / average))
result.append(score)
return result
return scores
def make_retriever(**options):
"""Return retrieve(question): the chunk ids that score above 0, best first."""
scores = make_scorer(**options)
def retrieve(question):
s = scores(question)
order = sorted(range(len(CHUNKS)), key=lambda i: (-s[i], i))
return [CHUNKS[i][0] for i in order if s[i] > 0]
return retrieve
def gold_ranks(retrieve, pairs):
"""Position of the gold chunk in the ranking for each question, or None when it is absent."""
ranks = []
for question, gold in pairs:
ranked = retrieve(question)
ranks.append(ranked.index(gold) + 1 if gold in ranked else None)
return ranks
def summary(ranks):
n = len(ranks)
hit = lambda k: sum(1 for r in ranks if r and r <= k)
mrr = sum(1 / r for r in ranks if r) / n
return f"hit@1 {hit(1):>2}/{n} hit@3 {hit(3):>2}/{n} hit@5 {hit(5):>2}/{n} MRR {mrr:.3f}"
def triage(retrieve, case, k=3):
question, gold, fact, answer = case
if gold is None:
return "not in the help center"
if gold not in retrieve(question)[:k]:
return "retrieval miss"
if fact.lower() not in answer.lower():
return "generation miss"
return "ok"
baseline = make_retriever()
question, gold = TUNING[0]
ranked = baseline(question)
print("Step 1 what did the retriever send for the jacket question?")
print(f" question: {question}")
print(f" query words: {tokens(question)}")
print(f" chunks that share a word: {ranked}")
print(f" gold chunk {gold}: {'rank ' + str(ranked.index(gold) + 1) if gold in ranked else 'never returned'}")
# Check one score by hand: 'back' is the only shared word, so refund-timing scores idf * tf / (tf + K1 * (...)).
lengths = [len(tokens(text)) for _, _, text in CHUNKS]
n, df = len(CHUNKS), sum(1 for _, _, text in CHUNKS if "back" in tokens(text))
idf = math.log(1 + (n - df + 0.5) / (df + 0.5))
average, length = sum(lengths) / n, lengths[2]
by_hand = idf * 1 / (1 + K1 * (1 - B + B * length / average))
print(f" by hand: idf {idf:.3f}, chunk length {length}, average {average:.3f}, score {by_hand:.3f}")
print(f" the scorer gives refund-timing {make_scorer()(question)[2]:.3f}")
print("Step 2 sort 11 recorded answers, top 3 chunks")
labels = Counter()
for case in CASES:
label = triage(baseline, case)
labels[label] += 1
print(f" {label:<23} {case[0]}")
print(f" {dict(labels)}")
print("Step 3 does a bigger k help? 10 questions with a gold chunk")
ranks = gold_ranks(baseline, TUNING)
for k in (3, 5):
print(f" k={k} {sum(1 for r in ranks if r and r <= k)} of 10 found")
for (q, _), r in zip(TUNING, ranks):
if r is None:
print(f" never found, {len(baseline(q))} chunk(s) returned at any k: {q}")
print("Step 4 one change at a time, 10 tuning questions")
for name, options in VARIANTS:
ranks = gold_ranks(make_retriever(**options), TUNING)
print(f" {name:<18} {summary(ranks)} gold ranks {' '.join(str(r or '-') for r in ranks)}")
print("Step 5 the same four retrievers on questions they were not built from")
for label, pairs in (("from the docs", FROM_THE_DOCS), ("customer words", CUSTOMER_WORDS)):
print(f" {label}")
for name, options in VARIANTS:
ranks = gold_ranks(make_retriever(**options), pairs)
print(f" {name:<18} {summary(ranks)} gold ranks {' '.join(str(r or '-') for r in ranks)}")
uv run --no-project --python 3.12 python rag_triage.pyStep 1 what did the retriever send for the jacket question?
question: How long do I have to send back a jacket?
query words: ['long', 'send', 'back', 'jacket']
chunks that share a word: ['refund-timing', 'damaged-item']
gold chunk returns-outdoor: never returned
by hand: idf 2.303, chunk length 14, average 14.429, score 1.060
the scorer gives refund-timing 1.060
Step 2 sort 11 recorded answers, top 3 chunks
retrieval miss How long do I have to send back a jacket?
ok When will my money come back after I return something?
generation miss Do you deliver to Canada and who pays the taxes?
ok Can I cancel the order I placed this morning?
retrieval miss Does the tent come with a guarantee?
retrieval miss My boots showed up broken, what now?
ok Can I use two discount codes?
ok How fast is express delivery?
ok I bought the wrong size, can I swap it?
ok Do gift cards expire?
not in the help center Will you match a lower price from another store?
{'retrieval miss': 3, 'ok': 6, 'generation miss': 1, 'not in the help center': 1}
Step 3 does a bigger k help? 10 questions with a gold chunk
k=3 7 of 10 found
k=5 7 of 10 found
never found, 2 chunk(s) returned at any k: How long do I have to send back a jacket?
never found, 0 chunk(s) returned at any k: Does the tent come with a guarantee?
never found, 0 chunk(s) returned at any k: My boots showed up broken, what now?
Step 4 one change at a time, 10 tuning questions
baseline hit@1 5/10 hit@3 7/10 hit@5 7/10 MRR 0.600 gold ranks - 1 1 2 - - 1 2 1 1
+ chunk title hit@1 7/10 hit@3 7/10 hit@5 7/10 MRR 0.700 gold ranks - 1 1 1 - - 1 1 1 1
+ synonyms hit@1 9/10 hit@3 9/10 hit@5 9/10 MRR 0.900 gold ranks - 1 1 1 1 1 1 1 1 1
+ plural stemming hit@1 10/10 hit@3 10/10 hit@5 10/10 MRR 1.000 gold ranks 1 1 1 1 1 1 1 1 1 1
Step 5 the same four retrievers on questions they were not built from
from the docs
baseline hit@1 5/6 hit@3 6/6 hit@5 6/6 MRR 0.917 gold ranks 2 1 1 1 1 1
+ chunk title hit@1 5/6 hit@3 6/6 hit@5 6/6 MRR 0.917 gold ranks 2 1 1 1 1 1
+ synonyms hit@1 5/6 hit@3 6/6 hit@5 6/6 MRR 0.917 gold ranks 2 1 1 1 1 1
+ plural stemming hit@1 6/6 hit@3 6/6 hit@5 6/6 MRR 1.000 gold ranks 1 1 1 1 1 1
customer words
baseline hit@1 1/6 hit@3 5/6 hit@5 5/6 MRR 0.444 gold ranks 3 2 - 2 1 3
+ chunk title hit@1 1/6 hit@3 5/6 hit@5 5/6 MRR 0.444 gold ranks 3 2 - 2 1 3
+ synonyms hit@1 1/6 hit@3 5/6 hit@5 5/6 MRR 0.444 gold ranks 3 2 - 2 1 3
+ plural stemming hit@1 2/6 hit@3 5/6 hit@5 5/6 MRR 0.556 gold ranks 1 2 - 2 1 3Where you will see this
The BM25 defaults and idf in the code are from the Lucene BM25Similarity page, and the term weight is the one in Robertson and Zaragoza's survey. The BEIR benchmark reports BM25 as a robust baseline across 18 datasets. We do not compare retrievers here.
The term RAG comes from Lewis et al., whose retriever was a dense vector index, so it is not a keyword scorer like ours. An Apple research page notes that in RAG the queries are short user questions while the stored keys are long, curated text. That is one way a question and its answer chunk can look very different.
On the generation side, the Sufficient Context paper reports that larger models often give a wrong answer instead of abstaining when the context is not enough. Our price match question has the same shape: no chunk answers it, and the recorded answer makes up a policy. The facet-level tracing paper says that on its two datasets hallucinations were driven less by retrieval accuracy and more by how the evidence was used. The Ragas paper splits a RAG pipeline into a retrieval part and a generation part and offers metrics for each, which is the idea of this guide. None of these papers gives a rate for your system, and neither does ours.
What this does not do
- The help center, the questions and the recorded answers are made up. The bot is recorded text, not a model, so we did not test any model. We say nothing about real RAG systems or about which failure is more common.
- The retriever is a keyword scorer written for this guide. We did not compare it with an embedding retriever, so we do not say which is better. Step 3 holds only for a scorer that returns just the chunks that share a word.
- The generation rule looks for one phrase in the answer. It is not a measure of answer quality. A real system needs a person or a checked judge to read the answers.
- The scorer uses the common BM25 defaults k1 = 1.2 and b = 0.75, which are not the right values for every index. With only 14 chunks the idf values are small and change a lot if you add one chunk.
- We chose the three changes after we saw the three misses, so 10 of 10 on the tuning questions is built in. Both new sets have 6 questions that we wrote after reading the chunks, so their numbers are rough.
- None of the 8 words in the word list appears in either new set, so we could not show whether the list helps on new questions. A list built from the misses you have seen covers only the words you have seen.
- We did not run lost in the middle. The paper is about long inputs, and ours are short.
- We show three kinds of wrong answer. We did not test chunk size, stale pages, rerankers or hybrid search.
- Papers, docs and code change. We read every source on the date below.
Next: keep the failing questions
Save each failing question as a test case, so a later change cannot break it without anyone noticing. Turn a failed agent run into an eval that blocks the release shows the steps for an agent: save the case, grade the result and let CI block the release. The same steps fit a list of retrieval questions.
To judge a change with only a few questions, read Did the fix work, or was it luck? Compare two versions of an LLM agent. Its test does not fit our numbers, because we chose the three changes after we saw the three misses. More guides are on the guides page.
Explore this topic in the workspace
Topic Breakdown gives you a structured revision guide for this topic and lets you highlight any phrase to open side-by-side chats with worked examples. Sign in to open it.
Open breakdownOr type RAG: evaluating retrieval and generation into the topic box on the home page. To see a lesson first, open the sample breakdown on Machine Learning.
Sources
All sources were checked on 16 November 2026. Read each page for the full context.
- Apache Lucene 10.3.0: BM25Similarity
Defaults k1 = 1.2 and b = 0.75, and idf implemented as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5)). BM25 was introduced in Okapi at TREC-3 in 1994. The page does not spell out the term frequency part, so we took it from the survey below.
- Robertson and Zaragoza: The Probabilistic Relevance Framework, BM25 and Beyond (2009)
Equations 3.13 to 3.15 give the classic BM25 term weight as tf divided by (k1 × ((1 - b) + b × dl / avdl) + tf), times a term weight w. Our code uses the idf above as that weight. Only the title page and the page with the formula were read.
- Lewis et al.: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv 2005.11401)
Where the term RAG comes from. Its non-parametric memory is a dense vector index of Wikipedia read by a neural retriever, so it does not describe our keyword scorer. Only the abstract was read.
- Joren et al.: Sufficient Context (arXiv 2411.06037)
Larger models answer well when the context is sufficient but often give a wrong answer instead of abstaining when it is not, and smaller models hallucinate or abstain even with sufficient context. Only the abstract was read.
- Elchafei et al.: Facet-Level Tracing of Evidence Uncertainty and Hallucination in RAG (arXiv 2604.09174)
On medical QA and HotpotQA, the abstract says hallucinations in RAG systems are driven less by retrieval accuracy and more by how retrieved evidence is integrated during generation. Only the abstract was read, and we cite no figure from the paper.
- Liu et al.: Lost in the Middle (arXiv 2307.03172)
Performance is often highest when the relevant information is at the beginning or end of the input and degrades when it is in the middle, even for long-context models. Only the abstract was read.
- Thakur et al.: BEIR (arXiv 2104.08663)
On 18 datasets, BM25 is a robust baseline, re-ranking and late-interaction models do best on average at high computational cost, and dense and sparse retrievers are cheaper but often underperform on new data. Only the abstract was read.
- Es et al.: Ragas (arXiv 2309.15217)
A RAG pipeline has a retrieval module and a generation module, and its metrics cover the retrieval system, how faithfully the model uses the passages, and the generation itself. Only the abstract was read.
- Apple Machine Learning Research: Faster Retrieval for Better, More Efficient RAG (5 November 2026)
In RAG, queries come from short user questions while keys are long, informative, curated text, so the two are substantially different. We read the first part of the page and use that one point.