Topics
Attention-based architectures that became the backbone of modern language and multimodal models.
Language Models · Google Research
BERT made deep bidirectional Transformer pretraining practical, letting one pretrained encoder be fine-tuned into strong task-specific NLP systems with minimal architecture changes.
Language Models · Google DeepMind
Chinchilla showed that many large language models were undertrained, and that better compute allocation can beat simply making parameters larger.
GPT-3 showed that a 175B autoregressive language model could perform many tasks from examples in the prompt, without gradient updates or task-specific fine-tuning.
Language Models · Google Research
PaLM used the Pathways system to train a 540B dense Transformer and showed how scale improves few-shot language, reasoning, and code performance.
Language Models · Google Research
T5 unified NLP transfer learning by casting every task as text input to text output, then systematically studying objectives, data, scale, and fine-tuning choices.
Vision Foundation Models · Google Research
ViT showed that a standard Transformer can compete in image recognition when images are split into patches and trained at sufficient scale.
Transformers · Google Research
The Transformer removed recurrence and convolution from sequence transduction, replacing them with attention and parallel training; almost every modern LLM stands on that move.
Efficient AI · Stanford University
FlashAttention keeps attention exact but makes it IO-aware, using tiling to reduce slow GPU memory traffic and make long-sequence Transformers faster and cheaper.