ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
Milestone Papers
ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

(2019-10) ZeRO by Microsoft

Paper
(2019-10) ZeRO by Microsoft

Relevant Sites

GLM-130B: An Open Bilingual Pre-trained Model

(2022-10) GLM-130B by Tsinghua

GPT-4 Technical Report

(2023-03) GPT 4 by OpenAI

PaLM: Scaling Language Modeling with Pathways

(2022-04) PaLM by Google

Language Is Not All You Need: Aligning Perception with Language Models

(2023-02) Kosmos-1 by Microsoft

Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling

(2023-04) Pythia by EleutherAI et al.

Using Deep and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model

(2022-01) Megatron-Turing NLG by Microsoft&NVIDIA

Leave a Reply

Your email address will not be published. Required fields are marked *