AI Glossary
Self-attention
What is Self-attention?
The core mechanism in transformer models that allows every position in a sequence to attend to every other position when computing its representation. Self-attention enables transformers to capture long-range dependencies that earlier architectures missed. It is computationally expensive — scaling quadratically with sequence length — which is why extending context windows is technically challenging.
Example in practice
When an AI correctly identifies that "the bank" in a financial document refers to a financial institution rather than a riverbank — based on context from earlier in the same paragraph — self-attention is what resolved that semantic ambiguity.
Learn more
See Self-attention applied in a professional context through this free course.
AI Fundamentals for Professionals →