Definition: activation function

A mathematical operation for a neuron in a neural network that determines the value it passes to the next layer. The activation function is applied after the nueron multiplies its inputs by the weights of their connections plus a bias.

Common Activation Functions
ReLU (Rectified Linear Unit) and Leaky ReLU are the most common. They help prevent "dead" neurons that no longer generate meaningful output. Sigmoid and Tanh (hyperbolic tangent) are less common, while Softmax is used in the output layer. See neural network.