The Transformer is a deep learning architecture introduced in the 2017 paper Attention Is All You Need by Vaswani et al. It serves as the foundation for modern large language models, including GPT, Claude, and Gemini, as well as vision and audio systems.
Key Concepts
- Self-Attention Mechanism: Traditional architectures like Recurrent Neural Networks (RNNs) processed words one by one, making it difficult to remember long-range context. Transformers evaluate an entire sequence simultaneously, assigning dynamic attention weights to capture relationships between words regardless of distance.
- Massive Parallelisation: Because the model does not depend on sequential step-by-step recurrence, training scales efficiently across thousands of GPUs.
- Positional Encodings: Since tokens are ingested concurrently rather than in order, mathematical position vectors are added to token embeddings so the model retains word order.
- Encoder-Decoder Design: The original design paired an Encoder (to analyse input sequences) with a Decoder (to generate output sequences). Modern variants often use decoder-only (e.g., GPT-style causal language models) or encoder-only (e.g., BERT-style representations) setups depending on the objective.