AI Glossary
Pre-training
What is Pre-training?
The initial large-scale training phase of a foundation model, where it learns general language patterns from vast amounts of text. Computationally very expensive — costing millions of dollars for frontier models. Pre-training creates the base capability; fine-tuning and prompting refine it.
Example in practice
When OpenAI spent an estimated $100M+ training GPT-4 on internet-scale text, that was pre-training — the one-time cost that produces the base model all subsequent products are built on.