AI Term of the Moment
Attention Is All You Need
Definition: neural network
The architecture of today's small and large language models, which provides the source of knowledge and intelligence that chatbots such as ChatGPT, Gemini and Claude use to generate answers.
Servers and GPUs
Data centers can have thousands of servers. The more servers, the larger the neural network, the larger the language model and the greater capacity to learn complex patterns and relationships. Desktop and mobile devices run the chatbots that communicate with the models in a data center; however, local devices are increasingly used to run small models and even train them; for example, see DGX Spark.
Layers and Nodes
A Single Neuron/Node
Tracing a Neuron
The neural network is a pattern detection system. The text used for training the network is turned into "tokens" (words, parts of words and punctuation). A token is converted into an "embedding," which is a one-dimensional array of numbers known as a "vector." Each layer transforms the embeddings mathematically and makes them increasingly useful for predicting the next token. See AI token and backpropagation.
A Tiny Network
A Larger Network
There Are Many Neural Network Designs
Servers and GPUs
Data centers can have thousands of servers. The more servers, the larger the neural network, the larger the language model and the greater capacity to learn complex patterns and relationships. Desktop and mobile devices run the chatbots that communicate with the models in a data center; however, local devices are increasingly used to run small models and even train them; for example, see DGX Spark.
A neural network is a "math machine" that learns from examples. It comprises multiple layers of computational units called "nodes" or "neurons" that are mathematically connected to each other (see diagrams below). There is an input layer, an output layer and any number of "hidden" layers in between. To learn how the network is trained, see backpropagation.
A neuron, or "node," is a mathematical processing unit within the network, and a single GPU processes the calculations for thousands of neurons simultaneously. The activation function determines if the value leaving the neuron should be altered. The weights and bias values are modified in the training passes when predictions are compared against the correct answers. If this all sounds very strange... it is! See weights and biases.
Tracing a Neuron
The neural network is a pattern detection system. The text used for training the network is turned into "tokens" (words, parts of words and punctuation). A token is converted into an "embedding," which is a one-dimensional array of numbers known as a "vector." Each layer transforms the embeddings mathematically and makes them increasingly useful for predicting the next token. See AI token and backpropagation.
This neural network example has only 17 neurons/nodes in three hidden layers and might be used for limited pattern recognition in a small sensor.
This neural network example has 8,192 neurons/nodes in eight hidden layers for a total of 65 thousand nodes and 4.3 billion connections that are adjusted in the training stages. Now imagine a neural network with 100 million nodes and you can begin to visualize the complexity of large AI models. For the neural network that is easier to grasp, see convolutional neural network.
This is a very small sample from the Asimov Institute in the Netherlands. Only a few layers are shown for each type, but there can be hundreds of layers between the input and output. (Images courtesy of Fjodor van Veen and Stefan Leijnen (2019). The Neural Network Zoo.)