Milestone Papers

Scaling Laws for Neural Language Models

(2020-01) Scaling Law by OpenAI

ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

(2019-10) ZeRO by Microsoft

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

(2019-10) T5 by Google

Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

(2019-09) Megatron-LM by NVIDIA

Language Models are Unsupervised Multitask Learners

(2019-02) GPT 2.0 by OpenAI

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

(2018-10) BERT by Google

Improving Language Understanding by Generative Pre-Training

(2018-06) GPT 1.0 by OpenAI

Attention Is All You Need

(2017-06) Transformers by Google


1 2