Using Deep and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
Milestone Papers
Using Deep and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model

(2022-01) Megatron-Turing NLG by Microsoft&NVIDIA

Paper
(2022-01) Megatron-Turing NLG by Microsoft&NVIDIA

Relevant Sites

Scaling Language Models: Methods, Analysis & Insights from Training Gopher

(2021-12) Gopher by DeepMind

Scaling Laws for Neural Language Models

(2020-01) Scaling Law by OpenAI

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

(2018-10) BERT by Google

Visual Instruction Tuning

(2023-04) LLaVA by UW–Madison&Microsoft

Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling

(2023-04) Pythia by EleutherAI et al.

Improving Language Understanding by Generative Pre-Training

(2018-06) GPT 1.0 by OpenAI

Leave a Reply

Your email address will not be published. Required fields are marked *