Using Deep and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
Milestone Papers
Using Deep and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model

(2022-01) Megatron-Turing NLG by Microsoft&NVIDIA

Paper
(2022-01) Megatron-Turing NLG by Microsoft&NVIDIA

Relevant Sites

DeepSeek-R1 91,457

(2025-1) DeepSeek-R1 by DeepSeek

Mamba: Linear-Time Sequence Modeling with Selective State Spaces

(2023-12) Mamba by CMU&Princeton

PaLM-E: An Embodied Multimodal Language Model

(2023-03) PaLM-E by Google

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

(2021-01) Switch Transformers by Google

Attention Is All You Need

(2017-06) Transformers by Google

Scaling Instruction-Finetuned Language Models

(2022-10) Flan-T5/PaLM by Google

Leave a Reply

Your email address will not be published. Required fields are marked *