The part of an AI chatbot or agent that runs the user's question against the trained model in order to generate answers. For example, ChatGPT runs one of the GPT models, Gemini uses one of the Gemini models. A peculiar name for sure; however, the inference term dates back to very early AI (see
expert system).
Also called "inference runtime" and "inference engine," the inference software uses the statistical patterns in the language model for its intelligence. For every output token (word, part of word, etc.) in the answer generated for the user, the inference engine has generally made a forward pass through the model (see
AI attention).
Three Parts to Answering Your Question
The back-and-forth dialog between you and the chatbot is handled by the chat software, which means a chatbot comprises the model, the chat software and the inference engine. See
long-horizon context and
AI training vs. inference.
Inference Is Constantly Being Improved
Although it can take months to train huge language models with trillions of parameters, enhancing inference processing is where the action is going forward. Why? Because millions of people may be using a chatbot at the same time, and every shortcut in inference processing means less data center time, less expense for the AI company and more user satisfaction. If cheaper prices are then passed on to users, a company's AI becomes more competitive.
Data Center and Desktop
The hardware that inference software uses is typically in a data center, but it can be in the user's own desktop computer. Increasingly, even small language models can be trained locally. However, large language models (LLMs) are always trained in a data center.
A Term with Wiggle Room
The English word "inference" implies assumption and conjecture. Apparently, in the early days of AI, "inferring" an answer seemed a safer bet than claiming to "generate" an answer, the latter implying a degree of accuracy. See
neural network and
deep learning.