Splitting text into meaningful units, such as words and numbers is called ________.
A) inflectionization
B) tokenization.
C) lemmatization.
D) parts-of-speech tagging

Question

Quizplus · Accepted Answer

Tokenization is the process of breaking down a text into meaningful units, such as words, phrases, and symbols, which are called tokens. Inflectionization is not a commonly used term in natural language processing. Lemmatization refers to reducing a word to its base or root form. Parts-of-speech tagging is the process of identifying the grammatical category of each word in a sentence.

Splitting Text into Meaningful Units, Such as Words and Numbers