Megatron-DeepSpeed
Training Frameworks
Megatron-DeepSpeed

DeepSpeed version of NVIDIA's Megatron-LM that adds additional support for several features such as MoE model training, Curriculum Learning, 3D Parallelism, and others.

GitHub
DeepSpeed version of NVIDIA's Megatron-LM that adds additional support for several features such as MoE model training, Curriculum Learning, 3D Parallelism, and others.

Relevant Sites

Transformer Engine 2,892

A library for accelerating Transformer model training on NVIDIA GPUs.

veRL 15,240

veRL is a flexible and efficient RL framework for LLMs.

Mesh Tensorflow 1,620

Mesh TensorFlow: Model Parallelism Made Easier.

GPT-NeoX 7,328

An implementation of model parallel autoregressive transformers on GPUs, based on the DeepSpeed library.

torchtune 5,581

A Native-PyTorch Library for LLM Fine-tuning.

NeMo Framework 16,065

Generative AI framework built for researchers and PyTorch developers working on Large Language Models (LLMs), Multimodal Models (MMs), Automatic Speech Recognition (ASR), Text to Speech (TTS), and Computer Vision (CV) domains.

Leave a Reply

Your email address will not be published. Required fields are marked *