Retrieval-Augmented Generation

A language model answers from its parameters, and its parameters hold what it was trained on. The library’s Saturday hours may not be in there. They may be in there as they stood some years ago. The model has no way to tell us which. What is missing is a way to put the current schedule in front of the model at the moment the question is asked.

That takes two jobs, and this part is about keeping them apart. Retrieval finds the passages that might hold the answer. Generation writes an answer from the question and those passages. They are kept separate because they fail differently. A retriever can hand over the wrong passage, and a model can misread the right one. If the two were a single step, a wrong answer would not say which of those had happened.

The collection on this page is three invented passages, each with a source number.

A collection needs passages that are small enough to retrieve and include in a prompt, but large enough to preserve the information an answer needs. A sentence split away from its heading or qualifying paragraph can become misleading.

Source 2 shows what is at stake. Its second sentence says that it closes at four. Kept beside the first sentence, that is the Saturday closing time. Stored as a passage of its own, it no longer says what closes or on which day. It also shares no word with a question such as “When does the library close on Saturday?”, so a retriever that matches words would never return it. The last practice problem makes that split and runs the question.

Keep the original source and location alongside each passage. A generated citation is useful only if it points to the material actually used and that material supports the claim. Updating the collection changes available evidence without necessarily changing model parameters.

The source number is what makes an answer checkable. An answer that says ten and cites Source 2 can be held against the sentence in Source 2. And because the schedule lives in the collection, a change of hours is an edit to one passage. No parameter of the model has to change.

The retriever ranks passages for the question. Exact term matching works when question and source use similar words. Learned embeddings can compare broader semantic relationships. Some systems combine lexical and vector retrieval or rerank initial candidates.

Ask the live collection the full question, “When does the library open on Saturday?”, and it scores all three passages. Part 2 shows how the scores are calculated.

Similarity is a selection signal, not proof of relevance. A passage about weekday hours can share many words with a Saturday question and still be the wrong evidence. Read the content, especially dates, conditions, and exceptions.

The table is that warning in numbers. The right passage is ranked first, and the weekday passage is not far behind it at 0.416, because it shares the words the, library and on with the question. One word, Saturday, separates the passage that answers this question from the passage that answers a different one. A retriever asked for two passages would hand over both, and it would then be up to the model to notice the difference in days.

The example case-folds text, extracts word-like pieces, and builds a vocabulary from three invented passages. Each text becomes a count vector. The query is compared against passage vectors using cosine similarity.

Each of those terms is a small idea. Case-folding treats capital and small letters as the same, so Saturday and saturday are one word. The vocabulary is the list of every different word in the passages, which comes to 22 words for this collection. A count vector has one entry for each vocabulary word, holding how many times that word occurs in the text. Cosine similarity measures how nearly two such vectors point the same way. It multiplies matching entries and adds them up, then divides by the lengths of the two vectors, so that a passage is not favored merely for holding more words.

For an illustrative two-word vocabulary, a query can exactly match one direction while partially matching another.

Vocabulary = [library, Saturday]
Query q = [1, 1]
Passage A = [1, 1]
Passage B = [1, 0]

cos(q, A) = (1 × 1 + 1 × 1) / (√2 × √2) = 1
cos(q, B) = (1 × 1 + 1 × 0) / (√2 × 1) = 1 / √2

The live collection contains more words, so its scores differ from these two-coordinate illustrations. Only positive matches are returned. If the query has no words in the indexed vocabulary, its vector has no length and the retriever returns no passages.

Here is the same calculation on the live collection, for the query the example opens with, library Saturday. The query vector has a one at library and a one at saturday, so its length is √2. Sources 1 and 2 each hold eleven words with the word at appearing twice, which gives each a squared length of 13. Source 3 holds ten different words, for a squared length of 10. The numerator of each score counts the shared words.

Source 2 shares library and saturday:  2 / (√2 × √13) ≈ 0.392
Source 3 shares library:               1 / (√2 × √10) ≈ 0.224
Source 1 shares library:               1 / (√2 × √13) ≈ 0.196

The Saturday passage is first, as hoped. Second place is the surprise. The passage about renewing books outranks the passage about weekday hours, although neither mentions Saturday and each shares only the word library with the query. It ranks higher because it is shorter, and cosine divides by length. At the example’s default of two passages, the context handed on would be the Saturday hours and the renewal instructions. Nothing in that second passage bears on the question, and its score had no way to say so.

Exact matching has a second limit, and the other searches the example suggests run into it.

Elephant returns nothing because no passage contains it, which is the honest result. Renew returns nothing for a worse reason. Source 3 is about renewing books, but the word in it is renewed, and to an exact matcher those are two different words. Learned embeddings are meant to close that gap. This baseline leaves it open so that it can be seen.

A complete RAG application combines the question with selected passages and instructions about using them. The language model then performs normal conditional generation on that prompt. A simple prompt structure is:

Task: Answer using the supplied sources.
If the sources do not provide the answer, say so.

Question: When does the library open on Saturday?

[Source 2]
The library opens at ten on Saturday. It closes at four.

Answer with supporting source identifiers:

Each line of the template has a job. The task line tells the model to answer from the sources and not from memory. The second line gives it something to say when the sources are silent. That matters because a word-matching retriever returns its best matches whether or not any of them holds the answer. The question is restated so the model knows what to look for. The passage arrives with its source number, so the answer can cite it, and the closing line asks for that citation.

This is a prompt template, not a measured model response. Instructions alone cannot guarantee source-faithful output. The application must evaluate whether the answer agrees with the passages and whether citations support their associated claims.

The SDK baseline returns source indices, text, and cosine scores. It keeps retrieval inspectable without introducing a hosted model or claiming that a generated answer has been verified.

from oop_ml.numpy.modern import TermOverlapRetriever

retriever = TermOverlapRetriever([
    "The library opens at nine on weekdays.",
    "The library opens at ten on Saturday.",
])
for passage in retriever.retrieve("library Saturday", limit=2):
    print(passage.index, passage.score, passage.text)

The snippet indexes two shortened passages and prints two lines. The Saturday sentence comes first with index 1 and a score near 0.535, and the weekday sentence follows with index 0 and a score near 0.267. Indices count from zero, so index 1 is the second passage in the list.

Those scores are higher than the 0.392 and 0.196 the live collection gave the same query and the same two opening sentences. The words shared are the same. The passages are shorter. Each holds seven different words, so the passage length in the denominator is √7 where the live passages had √13.

Saturday sentence:  2 / (√2 × √7) ≈ 0.535
Weekday sentence:   1 / (√2 × √7) ≈ 0.267

If the needed passage was not retrieved, changing the answer model may not fix the failure. Check source coverage, chunk boundaries, retrieval ranking, and the context limit. If the right passage was supplied but the answer was wrong, inspect the generation step and its instructions.

The live collection supplies two different retrieval failures. A search for renew retrieves nothing although the collection covers renewals, which is the retriever’s failure. A question about Sunday opening cannot be answered from any of the three passages, which is a gap in source coverage. No change to the retriever or to the model will supply a fact the collection does not hold. In both cases a more capable answer model would be working from the wrong context, or from none.

Source text is evidence to read, not permission to follow arbitrary instructions embedded in a document. An application should preserve that distinction when constructing prompts and handling external material.

RAG does not automatically make facts current. The collection must be maintained, and the answer must use the relevant version. It also does not remove the need to say when evidence is missing or contradictory.

The library makes the point about currency concrete. Suppose the Saturday hours change and Source 2 is not edited. The retriever will go on returning the old sentence with the same score, and a faithful model will go on citing it. Every step will have worked, and the answer will be wrong.