* Scholarly Communications *

Talking Tokens in Artificial Intelligence

In AI, especially in natural language processing (NLP), a token is a unit of text that the model processes. Tokens are the building blocks of input and output for language models.

So, what exactly Is a token?

A token can be:

• A word (e.g., "hello")
• A subword (e.g., "un", "break", "able" from "unbreakable")
• A character (in some models)
• Or even punctuation (e.g., ".", ",")

Most modern models, like those based on the Transformer architecture, use subword tokenization. This helps handle rare or unknown words more efficiently.

Tokens are how the model understands and generates language. For example:

• The sentence "I love pizza!" might be split into tokens like:

  ["I", "love", "pizza", "!"]
  

  Or, in subword form:

  ["I", "love", "piz", "za", "!"]
  

Each token is converted into a numerical representation (embedding) that the model can process.

• A token is a chunk of text (word, subword, or character).
• Models process text as sequences of tokens.
• Tokenization helps models handle language efficiently and flexibly.