(1) For crypto, see
AI crypto token.
(2) An AI token is a mathematical representation of text. When text is fed into an AI model for training, it is first turned into tokens. When a prompt is given to a chatbot to generate an answer, the prompt is turned into tokens, and when the inference software combined with the chatbot software generates the answer, the output tokens must be turned into text so people can read it. To learn how tokens are analyzed in the training phase, see
AI backpropagation.
Text to Tokens to Embedding Vectors
Software called a "tokenizer" turns the text into tokens, which are numbers that represent a word, fragment of a word, punctuation (, . !) or symbol ($ # -). Each token is turned into an array of numbers known as an "embedding vector." The vectors start out with random numbers but each training pass adjusts them to become more relevant. To deliver an answer, the detokenizer turns the generated tokens back into text for human consumption. See
AI transformer.
Image Tokens
Images may be turned into tokens as well and the tokens represent visual data. For example, an image may be broken into "patches" of 16x16 pixels, and each patch is converted into a numerical representation.
Example: "This is a token."
Using the GPT model as the tokenizer, the sentence is first broken into five tokens and then mapped to embedding vectors as follows. The transformer (the T in GPT) processes the embedding vectors using its learned weights and biases. The following is a brief sample, In a real model, there can be hundreds and thousands of numbers in the embedding vectors in order to represent the complex relationships in human languages.
Sentence: "This is a token."
Word Token ID Embedding Vector
"This" 1052 [0.25 -0.73 0.91 0.14 -0.52]
" is" 318 [0.61 0.22 -0.45 0.83 0.17]
" a" 257 [-0.34 0.76 0.12 0.59 0.41]
" token" 11241 [0.88 -0.15 0.63 0.29 -0.72]
"." 13 [0.19 0.42 -0.81 0.36 0.55]