Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
Milestone Papers
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

(2021-01) Switch Transformers by Google

Paper
(2021-01) Switch Transformers by Google

Relevant Sites

Scaling Laws for Neural Language Models

(2020-01) Scaling Law by OpenAI

Improving alignment of dialogue agents via targeted human judgements

(2022-09) Sparrow by DeepMind

LaMDA: Language Models for Dialog Applications

(2022-01) LaMDA by Google

Improving Language Understanding by Generative Pre-Training

(2018-06) GPT 1.0 by OpenAI

Emergent Abilities of Large Language Models

(2022-06) Emergent Abilities by Google

Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality

(2024-05) Mamba2 by CMU&Princeton

Leave a Reply

Your email address will not be published. Required fields are marked *