How an AI answers from a 500 page PDF without reading all of it. The steps, the words interviewers expect, and the answer to the question from the reel.
| # | Word | Meaning |
|---|---|---|
| 01 | Context window | How much text a model can read at once. |
| 02 | Chunk | A small piece of the document. |
| 03 | Embedding | Numbers that represent meaning. |
| 04 | Vector database | Stores vectors and finds the nearest ones. |
| 05 | Top-k | How many closest chunks you retrieve. |
| 06 | Chunk overlap | Shared text between neighbouring chunks. |
| 07 | Re-ranking | A second, more careful sort of the retrieved chunks. |
Retrieval-Augmented Generation. Instead of giving the model a whole document, you find the few most relevant pieces (retrieval), add them to the prompt with the question (augmented), and let the model write the answer from them (generation).
Some models now have context windows big enough for hundreds of pages, but sending everything for every question is slower and costs more, and models can miss details buried in very long inputs. RAG sends only what is needed.
A small piece of the document, often a few hundred tokens (roughly a paragraph or two). Chunk size is a trade-off: too small loses context, too big adds noise.
A list of numbers (a vector) that represents the meaning of a text. Texts about similar things get vectors that are close together.
Each model has its own number space. Comparing a question vector from model A with chunk vectors from model B is like comparing GPS coordinates from two different maps.
It stores the chunk vectors and quickly finds the ones closest to the question vector (nearest-neighbour search), usually with approximate methods so it stays fast over millions of chunks.
How many closest chunks you retrieve (k = 3 in the reel). Higher k gives more context but more noise and cost.
1) Chunk overlap: let neighbouring chunks share some text (e.g. 10–20%), so a sentence at the border appears whole in at least one chunk. 2) Neighbour expansion: when a chunk is retrieved, also add the chunk before and after it ("sentence window" / "parent document" retrieval). 3) Smarter chunking: split on headings, paragraphs or sentences, not a fixed character count. 4) Re-ranking: retrieve more (say 20), then re-rank and keep the best few.
Make a small test set of real questions with known answers. Measure if the right chunks are retrieved (retrieval quality) and if the final answer is correct and only uses the given sources (answer quality).
Tell the model to answer only from the provided chunks, say "I don't know" when they don't contain the answer, and cite the page or chunk it used.
Enough to understand the idea (no vector database, just numpy). Real apps use a vector database and a proper chunker.
# pip install sentence-transformers numpy
from sentence_transformers import SentenceTransformer
import numpy as np
model = SentenceTransformer("all-MiniLM-L6-v2") # embedding model
text = open("biology_notes.txt", encoding="utf-8").read()
size, overlap = 800, 150 # characters
chunks = [text[i:i+size] for i in range(0, len(text), size - overlap)]
vecs = model.encode(chunks, normalize_embeddings=True) # chunk -> numbers
q = "What is photosynthesis?"
qv = model.encode([q], normalize_embeddings=True)[0] # same model!
scores = vecs @ qv # cosine similarity
top3 = np.argsort(-scores)[:3] # retrieval
context = "\n---\n".join(chunks[i] for i in top3) # augmented
prompt = f"Answer only from this context:\n{context}\n\nQuestion: {q}"
print(prompt) # send this to Claude / any LLM -> generation