BM25 vs. Semantic Search: How Search Relevance Scoring Actually Works

Search relevance is the layer that decides which of the products matching a query actually show up first. Two very different techniques dominate how that scoring happens: BM25 and semantic relevance.

What Is BM25?

BM25 (Best Matching 25) is a widely used keyword-ranking algorithm. It scores how relevant a product is to a query based on how often the query's terms appear in the product's text, adjusted for two things: how long the product's text is, and how common each term is across the whole catalog — a rare term matching is worth more than a common one.

BM25 predates modern vector-based methods and remains a standard baseline for lexical, keyword-based relevance. It's fast, well understood, and precise for exact or near-exact queries — searching a specific model number or product name is exactly the case BM25 handles well.

What Is Semantic Relevance?

Semantic relevance scores products by similarity in meaning rather than by literal term overlap. Both the query and the product data are embedded into a shared vector space, and relevance is calculated as distance or similarity between those embeddings.

This is what allows a descriptive or paraphrased query — "something elegant for a dinner party" — to rank a matching product sensibly, even when none of the query's words appear anywhere in that product's title or description. BM25 has no mechanism for this at all: if the literal terms aren't present, there's nothing for it to score.

BM25 vs. Semantic Search: A Direct Comparison

• Factor by factor:
• Matches on: BM25 scores literal term overlap; semantic search scores meaning and similarity.
• Best for: BM25 handles exact or near-exact queries well; semantic search handles descriptive, paraphrased queries.
• Typos and synonyms: BM25 handles these poorly without extra rules; semantic search handles them natively in most cases.
• Speed: BM25 is very fast; semantic search carries slightly higher overhead.
• Maturity: BM25 is decades-old and well understood; semantic search is newer and still evolving.

Neither approach is strictly better — they fail on different kinds of queries, which is exactly why most production systems don't pick one.

Why Most Systems Use Both

Hybrid search runs BM25 and semantic scoring together, then blends the two into one final ranking value, often with additional business signals — popularity, margin, recency — layered on top. This keeps the speed and precision of keyword matching for simple, exact queries, while still catching the complex, natural-language ones that BM25 alone would miss entirely.

In practice, BM25 isn't obsolete just because semantic search exists. Many hybrid systems keep it as one of several blended signals rather than replacing it outright, because it's still the most precise, cheapest signal available for the queries it's suited to.

How to Tell If Your Relevance Scoring Is Actually Good

Two checks matter, and neither is sufficient alone. Human relevance judgments — rating a sample of query-result pairs directly against a rubric — catch cases where results look wrong even if the system ran without errors. Behavioral proxies, like click-through rate and add-to-cart rate on top results, catch the inverse: a technically correct-looking result set that shoppers still don't act on.

A relevance model can run cleanly and still return results nobody clicks. That's why teams combine both checks rather than relying on system health metrics alone.

The Bottom Line

BM25 scores relevance by literal term overlap; semantic relevance scores it by meaning. Most modern ecommerce search systems blend both rather than choosing one, because they fail on different, complementary kinds of queries — and testing relevance quality means checking both how results look and whether shoppers actually act on them.

Want to see how hybrid relevance scoring performs on your own catalog? Book a demo to try it against your real queries.