Tokenization
Tokenization is the process of splitting text — a search query or a product title — into smaller units (tokens), usually words or sub-words, before a keyword search engine indexes or matches them.
Part of Search Infrastructure
How a system tokenizes affects what counts as a match: whether "wireless-earbuds" splits into one token or two, or whether plurals and punctuation are normalized, changes which queries a product shows up for. It happens before the inverted index is built, on both catalog data and incoming queries.