Topics
Models trained to understand, generate, and transform natural language at scale.
Language models are the center of modern AI infrastructure because they turn text into a general interface for reasoning, retrieval, coding, agents, and multimodal systems. The important research thread is not a single model family, but the sequence of design choices that made scale useful: bidirectional pretraining, decoder-only few-shot learning, instruction following, compute-optimal training, and open model releases.
For SEO and learning, this topic is best read as a map of capability shifts. BERT made pretrained encoders practical for language understanding. GPT-3 made in-context learning visible. InstructGPT showed why human preference data matters. Chinchilla corrected the field's intuition about data and compute. Llama-style open models made language modeling a deployable ecosystem rather than only a closed frontier race.
Language Models · Google Research
BERT made deep bidirectional Transformer pretraining practical, letting one pretrained encoder be fine-tuned into strong task-specific NLP systems with minimal architecture changes.
Language Models · Google DeepMind
Chinchilla showed that many large language models were undertrained, and that better compute allocation can beat simply making parameters larger.
GPT-3 showed that a 175B autoregressive language model could perform many tasks from examples in the prompt, without gradient updates or task-specific fine-tuning.
Language Models · Google Research
BERT made deep bidirectional Transformer pretraining practical, letting one pretrained encoder be fine-tuned into strong task-specific NLP systems with minimal architecture changes.
Language Models · Google Research
T5 unified NLP transfer learning by casting every task as text input to text output, then systematically studying objectives, data, scale, and fine-tuning choices.
GPT-3 showed that a 175B autoregressive language model could perform many tasks from examples in the prompt, without gradient updates or task-specific fine-tuning.
InstructGPT showed that human preference data and RLHF could make smaller models more helpful and aligned than much larger raw language models.
Code Generation · Google DeepMind
AlphaCode attacked programming contests by generating many candidate programs, filtering them, and selecting diverse solutions likely to pass hidden tests.
Language Models · Google Research
BERT made deep bidirectional Transformer pretraining practical, letting one pretrained encoder be fine-tuned into strong task-specific NLP systems with minimal architecture changes.
Language Models · Google DeepMind
Chinchilla showed that many large language models were undertrained, and that better compute allocation can beat simply making parameters larger.
Code Llama adapts Llama-family models for code generation, infilling, and instruction-following, giving the open ecosystem stronger coding baselines.
GPT-3 showed that a 175B autoregressive language model could perform many tasks from examples in the prompt, without gradient updates or task-specific fine-tuning.
InstructGPT showed that human preference data and RLHF could make smaller models more helpful and aligned than much larger raw language models.
Code Generation · Google DeepMind
AlphaCode attacked programming contests by generating many candidate programs, filtering them, and selecting diverse solutions likely to pass hidden tests.
Language Models · Google Research
BERT made deep bidirectional Transformer pretraining practical, letting one pretrained encoder be fine-tuned into strong task-specific NLP systems with minimal architecture changes.
Language Models · Google DeepMind
Chinchilla showed that many large language models were undertrained, and that better compute allocation can beat simply making parameters larger.
Code Llama adapts Llama-family models for code generation, infilling, and instruction-following, giving the open ecosystem stronger coding baselines.
GPT-3 showed that a 175B autoregressive language model could perform many tasks from examples in the prompt, without gradient updates or task-specific fine-tuning.
InstructGPT showed that human preference data and RLHF could make smaller models more helpful and aligned than much larger raw language models.
Language Models · Google Research
PaLM used the Pathways system to train a 540B dense Transformer and showed how scale improves few-shot language, reasoning, and code performance.
Language Models · Google Research
T5 unified NLP transfer learning by casting every task as text input to text output, then systematically studying objectives, data, scale, and fine-tuning choices.