ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
Milestone Papers
ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

(2019-10) ZeRO by Microsoft

Paper
(2019-10) ZeRO by Microsoft

Relevant Sites

Scaling Laws for Neural Language Models

(2020-01) Scaling Law by OpenAI

PaLM 2 Technical Report

(2023-05) PaLM 2 by Google

Improving language models by retrieving from trillions of tokens

(2021-12) Retro by DeepMind

Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality

(2024-05) Mamba2 by CMU&Princeton

Training language models to follow instructions with human feedback

(2022-03) InstructGPT by OpenAI

PaLM-E: An Embodied Multimodal Language Model

(2023-03) PaLM-E by Google

Leave a Reply

Your email address will not be published. Required fields are marked *