Semantic Search and Retrieval-Augmented Generation
Adapted from: Jay Alammar & Maarten Grootendorst, Hands-On Large Language Models (O'Reilly, 2024), Chapter 8 — rewritten and condensed for teaching, with code examples adapted from the original.
Lesson 7 ended on a real limitation: every technique there still depended on whatever the LLM happened to memorize during training, plus whatever a tool call returned in the moment. This lesson tackles that gap directly with search — one of the first places transformer-based language models saw genuine industry-wide adoption. Encoder models like BERT famously drove some of the largest single-year quality jumps major search engines had ever shipped, precisely because they enabled semantic search: matching by meaning, not just overlapping keywords.
On a separate track, once people started treating chat-style LLMs like question-answering machines, a new problem showed up: models answer fluently and confidently, but not always correctly. This is the well-known hallucination problem, and the leading mitigation is retrieval-augmented generation (RAG) — giving the model real, retrieved information to ground its answer in, rather than relying purely on what it memorized.
Three Ways Language Models Upgrade Search
| Category | What it does |
|---|---|
| Dense retrieval | Embeds a query and a document archive into the same vector space, then returns the archive entries whose embeddings land closest to the query's |
| Reranking | Takes an existing shortlist of results (from any earlier search step) and rescoring/reordering them by relevance to the query |
| RAG | Adds a generation step after retrieval — the LLM reads the retrieved documents and formulates an answer, ideally citing its sources |
These three aren't mutually exclusive — a realistic pipeline commonly chains a fast first-stage retriever, a reranker to refine the shortlist, and (for RAG systems) a generation step at the end. The rest of this lesson works through each in turn.
Check: the three categories
Dense retrieval, reranking, and RAG
1 / 2What is the key structural difference between a dense retrieval system and a reranker?
Pick an answer to see if you are right.
Dense Retrieval
Recall from Lesson 2 that embeddings place text as points in a vector space, where nearby points represent similar meaning. Dense retrieval turns this directly into a search engine: embed the query into that same space, then return whichever archive entries sit closest to it.
Two questions come up almost immediately with this setup:
- Should a distant result even be returned? That's a system-design choice — many pipelines apply a maximum-distance cutoff so a query with genuinely no good match in the archive doesn't return misleadingly confident-looking "top results."
- Are a query and its best answer always similar text? Not necessarily — a question and its correct answer can look quite different on the surface. This is exactly why retrieval-oriented embedding models are specifically finetuned on (query, relevant document) pairs, rather than just reused off the shelf (more on this shortly).
A dense retrieval walkthrough
Here's the shape of a minimal dense retrieval pipeline, using Cohere's embedding API and FAISS for the index. Suppose we have a short, several-sentence synopsis of a documentary film as our searchable archive:
import cohere
import numpy as np
import pandas as pd
# Create and retrieve a Cohere API key from os.cohere.ai
co = cohere.Client(api_key="YOUR_KEY_HERE")
text = """
Abyssal Blue is a 2021 documentary directed by Mara Solano.
It follows a team of marine biologists exploring a newly discovered trench in the Pacific Ocean.
The crew used a custom-built submersible designed to withstand extreme pressure.
Filming took place over eighteen months using specialized deep-sea cameras.
The documentary features previously unseen footage of bioluminescent creatures.
Critics praised its cinematography and its scientific accuracy.
The film was made on a budget of two million dollars and earned twelve million dollars at the box office.
It won Best Documentary Feature at the Pacific Film Festival.
"""
# Split into a list of sentences and clean up whitespace
texts = [t.strip(" \n") for t in text.split(".") if t.strip(" \n")]
Embedding every chunk turns each sentence into a vector:
response = co.embed(texts=texts, input_type="search_document").embeddings
embeds = np.array(response)
print(embeds.shape)
# (8, 4096) -- 8 sentences, each embedded into a 4096-dimensional vector
A FAISS index makes nearest-neighbor lookup fast, even at large scale:
import faiss
dim = embeds.shape[1]
index = faiss.IndexFlatL2(dim)
index.add(np.float32(embeds))
def search(query, number_of_results=3):
query_embed = co.embed(texts=[query], input_type="search_query").embeddings[0]
distances, similar_item_ids = index.search(np.float32([query_embed]), number_of_results)
results = pd.DataFrame({
"texts": np.array(texts)[similar_item_ids[0]],
"distance": distances[0],
})
return results
search("how much money did the film make")
The top result would be the budget/box-office sentence — even though it shares almost no exact words with the query — because the meaning ("money," "earned," "budget") lines up in embedding space. A pure keyword search (using something like BM25, a standard lexical-search algorithm) would instead favor whichever sentence happens to share literal words with the query, which can easily miss the semantically right answer entirely — one of the clearest illustrations of what "semantic" search actually buys you over keyword matching.
Check: dense retrieval
Dense retrieval basics
1 / 2Why might a keyword search (like BM25) fail to find the best answer to a query, even when dense retrieval succeeds?
Pick an answer to see if you are right.
Caveats of Dense Retrieval
A few real limitations are worth knowing before relying on dense retrieval alone:
- No answer exists in the archive. The system still returns its nearest points — just with much larger distances. A threshold (or tracking which results users actually click and find useful) helps here, but there's no way to be certain a query has no good answer.
- Exact-phrase matching. If a user wants a literal phrase match, that's exactly what keyword search is built for — which is why hybrid search (combining dense retrieval and keyword search) is generally recommended over relying on dense retrieval alone.
- Domain mismatch. A retrieval model trained mostly on general web/Wikipedia-style text will often underperform in a narrow, different domain (legal documents, for instance) unless it saw enough of that domain during training.
- Answers that span multiple sentences. Everything so far assumed one sentence = one complete answer. Real answers often need more context than a single sentence provides — which raises the question of how to best break long documents into searchable pieces.
Chunking Long Texts
Transformer models have a fixed context length, so a long document can't just be embedded whole. Two broad strategies:
One vector per document. Represent the entire document with a single embedding — either by embedding just a representative slice (like the title or opening paragraph) or by averaging the embeddings of all its chunks. Both are fast to set up but lossy: the first ignores most of the document's content, and averaging compresses everything into one vector that blurs together whatever distinct ideas the document actually contains.
Multiple vectors per document. Split the document into smaller chunks and index each chunk's embedding separately. This gives full coverage and lets individual concepts get their own, more precise vector — generally the better approach for anything beyond a quick demo. Common chunking strategies include:
- One sentence per chunk — precise, but sometimes too granular to capture enough surrounding context.
- One paragraph per chunk (or roughly 3–8 sentences) — a reasonable default for most prose.
- Adding context to each chunk — prepending the document's title, or including some text from just before/after the chunk (overlapping chunks), so a chunk near a boundary doesn't lose meaning it depends on.
There's no universally "correct" chunk size — it depends on your documents and the kinds of questions you expect.
Check: dense retrieval caveats and chunking
Caveats and chunking strategies
1 / 3Why is hybrid search (dense retrieval + keyword search) generally recommended over dense retrieval alone?
Pick an answer to see if you are right.
Nearest Neighbor Search vs. Vector Databases
Once a query is embedded, finding its nearest archive vectors is just a distance calculation. At a small scale (thousands, maybe tens of thousands of vectors), brute-force distance computation with something like NumPy is entirely reasonable:
🧪 Brute-force nearest-neighbor search with NumPy
Once an archive grows into the millions of vectors, brute force stops being practical, and two categories of tools take over:
- Approximate nearest neighbor (ANN) libraries like Annoy or FAISS trade a small amount of accuracy for dramatic speedups — retrieving from a massive index in milliseconds, sometimes with GPU or multi-machine scaling.
- Vector databases like Weaviate or Pinecone go further: they let you add or delete vectors without rebuilding the whole index, and typically add filtering and other query features beyond plain vector distance.
Finetuning Embedding Models for Retrieval
Just as Lesson 4 covered finetuning a representation model for classification, embedding models can be finetuned specifically for retrieval — using (query, relevant document) pairs as positive examples and (query, irrelevant document) pairs as negative examples. Before finetuning, a query and several candidate documents might all sit at roughly the same distance in embedding space (they're all topically related, after all); finetuning pulls genuinely relevant pairs closer together while pushing irrelevant ones apart, teaching the model what "relevant" specifically means for your data.
Check: nearest neighbor search and finetuning
ANN search, vector databases, and retrieval finetuning
1 / 2When does it make sense to move from brute-force distance calculation to an ANN library like FAISS?
Pick an answer to see if you are right.
Reranking
Many organizations already have a working search pipeline and don't want to rebuild it from scratch. Reranking is often the easiest way to bolt language-model quality onto an existing system: add one more step at the end that reorders a shortlist of results by relevance.
A reranker takes a query and a set of candidate documents (from any earlier retrieval step — keyword, dense, or hybrid) and returns a relevance score for each. Continuing the earlier documentary example, reranking the shortlist for "how much money did the film make" would score the budget/box-office sentence clearly highest, with everything else trailing well behind — exactly the reordering a good reranker is supposed to produce, no training or tuning required on your end since it's a ready-to-use API/model.
In practice, an index might hold millions of entries, far too many to rerank directly — so a fast first-stage retriever (keyword, dense, or hybrid) shortlists perhaps a hundred or a thousand candidates, and only those get passed to the reranker. On a multilingual retrieval benchmark (MIRACL), adding a reranking step after a first-stage retriever has been shown to boost scores from roughly 36.5 to 62.8 (measured with nDCG@10, an evaluation metric covered later in this lesson) — a substantial jump from one extra pipeline stage.
How reranking models work
The dominant approach (sometimes called monoBERT, after an influential paper on the technique) treats reranking as a cross-encoder classification problem: the query and one candidate document are fed into the model together, letting it directly compare the two texts before outputting a single relevance score between 0 and 1. This should feel familiar — it's structurally the same classification setup covered in Lesson 4, just applied to a (query, document) pair instead of a single document.
If you want to run retrieval and reranking locally rather than through a hosted API, the open-source Sentence Transformers library documents exactly this workflow under its "Retrieve & Re-Rank" guide.
Check: reranking
Reranking
1 / 3Why is reranking often the easiest way for an existing search system to gain LLM-quality improvements?
Pick an answer to see if you are right.
Retrieval Evaluation Metrics
Semantic search borrows its evaluation methodology from the broader Information Retrieval field. Evaluating a search system needs three ingredients: a text archive, a set of test queries, and relevance judgments — a record of which archive documents are actually relevant to each test query.
Average precision
Consider a query with exactly one relevant document in the archive. If a system places it at position 1, that's a perfect result: precision at position 1 is 1/1 = 1.0. If instead the system buries it at position 3 (behind two irrelevant results), the score drops — the system gets penalized for making the user look further down the list before finding the right answer.
With multiple relevant documents for one query, the process is: at every position where a relevant document appears, compute precision-so-far (relevant docs seen / position), then average those per-position precision values together. That average is the query's average precision (AP).
Mean average precision (MAP)
Averaging the AP scores across every query in your test suite gives mean average precision (MAP) — a single number for comparing search systems head-to-head. (The name is a small in-joke: "mean average precision" is technically "the mean of the average precision scores," which is why it isn't just called "average precision.")
🧪 Computing average precision and MAP
A related, more nuanced metric is normalized discounted cumulative gain (nDCG), which handles graded relevance — where one document can be labeled more relevant than another, rather than a strict relevant/not-relevant split.
Check: MAP and evaluation
Average precision and MAP
1 / 2For a query with one relevant document, why does placing it at position 3 (behind two irrelevant results) score lower than placing it at position 1?
Pick an answer to see if you are right.
Retrieval-Augmented Generation (RAG)
RAG adds a generation step after retrieval: the retrieved documents get handed to an LLM along with the original question, and the LLM formulates an answer grounded in that retrieved context — ideally citing exactly which source backs each claim. This is what powers "chat with my data" use cases (grounding a model on internal company documents, or a specific book or dataset), and it's increasingly built directly into search engines themselves (Perplexity, Bing AI's chat features, and Google's AI-generated search summaries all use some version of this pattern).
Grounded generation with a hosted API
Continuing the earlier retrieval example, adding a generation step on top looks like this with Cohere's chat endpoint:
query = "how much did the film earn"
# 1. Retrieval (ideally hybrid, kept simple here)
results = search(query)
# 2. Grounded generation
docs_dict = [{"text": text} for text in results["texts"]]
response = co.chat(message=query, documents=docs_dict)
print(response.text)
A hosted chat endpoint like this can return not just the answer text, but citation spans — which exact substring of the answer is backed by which retrieved document — letting an application highlight or link back to sources directly in the UI.
RAG with fully local models
The same pattern works entirely offline, trading away citation spans (and some raw quality, if using a small local model) for zero API dependency. The pieces:
from langchain import LlamaCpp
# A quantized local model, exactly as covered in Lesson 7
llm = LlamaCpp(
model_path="Phi-3-mini-4k-instruct-fp16.gguf",
n_gpu_layers=-1,
max_tokens=500,
n_ctx=2048,
seed=42,
verbose=False,
)
from langchain.embeddings.huggingface import HuggingFaceEmbeddings
from langchain.vectorstores import FAISS
embedding_model = HuggingFaceEmbeddings(model_name="thenlper/gte-small")
db = FAISS.from_texts(texts, embedding_model)
A prompt template is what actually wires the retrieved context into the model's input:
from langchain import PromptTemplate
template = """<|user|>
Relevant information:
{context}
Provide a concise answer to the following question using the relevant information provided above:
{question}<|end|>
<|assistant|>"""
prompt = PromptTemplate(template=template, input_variables=["context", "question"])
from langchain.chains import RetrievalQA
rag = RetrievalQA.from_chain_type(
llm=llm,
chain_type="stuff",
retriever=db.as_retriever(),
chain_type_kwargs={"prompt": prompt},
)
rag.invoke("How much did the film earn?")
The "stuff" chain type simply "stuffs" every retrieved chunk into the {context} slot of the prompt — the simplest possible way to combine retrieval and generation, and a fine default until the retrieved context starts overflowing the model's context window.
Check: RAG basics
Retrieval-augmented generation
1 / 3What does "grounded generation" mean in the context of RAG?
Pick an answer to see if you are right.
Advanced RAG Techniques
Basic RAG (embed the question, retrieve, generate) breaks down in a few realistic situations. Several extensions address specific gaps:
- Query rewriting. A chatbot user's raw message is often too verbose, conversational, or dependent on earlier chat history to work well as a search query directly. An LLM can rewrite a rambling message into a focused search query before retrieval ever runs — some hosted chat APIs even expose a dedicated query-rewriting mode for exactly this.
- Multi-query RAG. Some questions genuinely need more than one search. "Compare this company's revenue in 2020 versus 2023" is better served by two separate targeted queries (one per year) than a single query hoping to find one document with both numbers. A smart query rewriter can also recognize when no search is needed at all, if it's confident it already knows the answer.
- Multi-hop RAG. Some questions require a sequence of dependent searches, where the second search depends on what the first one returned — e.g., first find which companies are relevant, then run a separate follow-up search for each one individually.
- Query routing. If an organization has multiple distinct data sources (an HR system, a CRM, internal docs), the model can be given the ability to decide which source to search based on the nature of the question, rather than searching everything indiscriminately.
- Agentic RAG. Push this far enough and retrieval starts looking like the agent pattern from Lesson 7 — each data source becomes a tool the model can choose to use, decide how to use, and potentially even write back to (not just read from). This requires meaningfully strong models to pull off reliably; as of this writing, it's mostly the largest, most capable models that handle this well.
Check: advanced RAG techniques
Advanced RAG techniques
1 / 3Why is query rewriting especially important for a chatbot-style RAG system, as opposed to a single-shot search box?
Pick an answer to see if you are right.
RAG Evaluation
Evaluating RAG systems is still an active area of development, but human evaluations commonly look at four axes:
- Fluency — is the generated text cohesive and well-written?
- Perceived utility — is the answer actually helpful and informative?
- Citation recall — what fraction of factual claims in the answer are backed by a citation that actually supports them?
- Citation precision — what fraction of the citations given actually support the claim they're attached to?
Human evaluation remains the gold standard, but it's slow and expensive to run repeatedly. LLM-as-a-judge approaches — using a capable LLM to score generations along these same axes — are increasingly used to automate this. The Ragas library implements this approach and adds a couple of its own useful metrics:
- Faithfulness — is the answer actually consistent with the retrieved context it was supposedly grounded in?
- Answer relevance — how directly does the answer address the question that was actually asked?
Check: RAG evaluation
Evaluating RAG systems
1 / 2What is the difference between citation recall and citation precision in RAG evaluation?
Pick an answer to see if you are right.
Chapter Summary
- Dense retrieval embeds queries and documents into the same vector space and returns whichever archive entries land nearest the query — powerful for meaning-based matches, but still needs a plan for exact-phrase queries (hybrid search), out-of-domain data, and answers that span more than one chunk.
- Chunking strategy matters: multiple vectors per document (one per chunk, sized to the content, sometimes overlapping for context) generally beats compressing an entire document into one vector.
- At small scale, brute-force nearest-neighbor search is fine; ANN libraries (FAISS, Annoy) and vector databases (Weaviate, Pinecone) take over as an archive grows into the millions.
- Rerankers bolt onto an existing search pipeline as a final step, using cross-encoder scoring (query and document processed together) to meaningfully improve a first-stage shortlist's ordering.
- Mean average precision (MAP) — the mean of each query's average precision — is a standard way to score and compare retrieval systems; nDCG extends this to graded (not just binary) relevance.
- RAG adds a generation step after retrieval, grounding the LLM's answer (and ideally its citations) in real retrieved documents rather than pure memorized knowledge — with query rewriting, multi-query, multi-hop, routing, and agentic variants handling progressively more complex information needs.
- RAG evaluation spans fluency, perceived utility, citation recall, and citation precision for human evaluation, plus faithfulness and answer relevance from automated LLM-as-a-judge tools like Ragas.
Text isn't the only modality language models are learning to reason about. The next lesson turns to multimodal models — systems that work with images alongside (or instead of) text.