Definition: backpropagation

The method used to train a large language model (LLM). An AI model's neural network learns patterns in training data by repeatedly adjusting its numerical parameters to improve its predictions.

Step 1 - Forward and Compare
Training starts by converting a batch of text into tokens with a numerical token ID and an embedding table initialized with random numbers for that batch. In the first forward pass, the token IDs retrieve the embedding vectors that are the actual values that pass through the neural network. See AI token.

The forward pass predicts the next tokens, and the predictions are compared with the actual next tokens. The difference between the predicted and actual next token is known as the "loss function," and the "cost function" is the average loss for the batch of training examples.

Step 2 - Backward and Adjust
Using this error information, backpropagation goes backward into the network and with the use of a "gradient descent" algorithm increases or decreases the neuron's weights and biases and the embedding table to generate a better result the next time around. A backpropagation pass generally follows a forward pass.

Prior to backpropagation, training was extremely slow, and backpropagation algorithms are considered one of the major breakthroughs in training AI models. AI engineers designing gradient backpropagation algorithms require a strong calculus background. See weights and biases.

Quadrillions and Quintillions of Changes
The forward and backward passes are repeated over and over until the model achieves the desired outcome. Very large language models that have billions of tokens can perform the forward pass and backpropagation steps millions of times, which means the total number of changes to the nodes and tables can be in the quadrillions (see space/time).

The Model Does Not Contain the Data
A neural network model is not a database of facts and figures. It does not contain actual data. It contains billions of learned numerical values that encode patterns and relationships of data. For all intents and purposes, considering the number of people involved in creating and using it, the neural network is the most unbelievable data processing activity ever conceived by mankind.

Adjust the Weights and Biases

The backpropagation algorithms constantly adjust the mathematical values between all the neurons. In a large language model, there can be billions and trillions of neurons. See neural network.