In natural language processing (NLP), what is the process of converting words or phrases into numerical vectors called?

The process of converting words or phrases into numerical vectors is most commonly referred to as embedding. In NLP, embeddings are representations of words or phrases that capture their meanings in a continuous vector space, allowing for various machine learning applications. This transformation enables algorithms to process textual data efficiently by converting it into a format that can be easily manipulated mathematically.

Vectorization is related and often used interchangeably with embedding in casual discussions, but it generally refers to the broader concept of transforming data (including text) into numerical forms. However, embedding specifically focuses on the technique of learning those representations from the context in which words appear.

Normalization pertains to adjusting values measured on different scales to a notionally common scale, which is not focused specifically on text representation. Tokenization, on the other hand, involves breaking down text into individual units or tokens, such as words or phrases, before any numerical transformation occurs. Therefore, the correct term for the process that directly refers to the conversion of words into numerical vector representations is embedding.