topic: RAG Retrieval, Step by Step
BM25 Search
How BM25 uses term frequency, inverse document frequency, and document length for keyword retrieval, its strengths with exact identifiers, and a practical LangChain example.
Table of contents
Introduction
BM25 Search is a commonly used retrieval method for finding relevant information based on keywords.
The main idea is simple:
If important words from the user's query also appear in a document chunk, that chunk is probably relevant.
For example, suppose our knowledge base contains:
Section 15.2 describes the requirements for employee termination.
Section 15.3 describes the requirements for temporary employee termination.
A user asks:
What does Section 15.2 say?
BM25 can give the first chunk a high score because the exact term:
15.2
appears in both the query and the document.
This makes BM25 particularly useful when the user's query contains important exact words, names, numbers, or identifiers.
How It Works
BM25 does not convert text into vector embeddings.
Instead, it looks at the words in the user's query and checks how those words appear across the document chunks.
Conceptually:
User Query
↓
Tokenize Words
↓
Compare Query Terms with Each Chunk
↓
Calculate BM25 Score
↓
Rank Chunks
↓
Top-k Results
BM25 mainly considers three ideas.
Term Frequency
If words from the query appear inside a document, that document becomes more relevant.
For example:
Query:
machine learning engineer
Document A:
machine learning engineer responsibilities...
Document B:
marketing manager responsibilities...
Document A will receive a higher score because more of the query terms appear in the document.
However, repeating the same word many times should not make a document infinitely more relevant.
For example:
machine machine machine machine machine...
BM25 uses a saturation mechanism so that repeated occurrences provide less and less additional benefit.
Inverse Document Frequency
Some words are more useful than others.
Suppose a word appears in almost every document:
employee
company
system
These words are not very useful for identifying one specific document.
However, something like: