ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
Milestone Papers
ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

(2019-10) ZeRO by Microsoft

Paper
(2019-10) ZeRO by Microsoft

Relevant Sites

LaMDA: Language Models for Dialog Applications

(2022-01) LaMDA by Google

Evaluating Large Language Models Trained on Code

(2021-08) Codex by OpenAI

PaLM 2 Technical Report

(2023-05) PaLM 2 by Google

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

(2021-01) Switch Transformers by Google

Improving alignment of dialogue agents via targeted human judgements

(2022-09) Sparrow by DeepMind

PaLM-E: An Embodied Multimodal Language Model

(2023-03) PaLM-E by Google

Leave a Reply

Your email address will not be published. Required fields are marked *