An implementation of model parallel autoregressive transformers on GPUs, based on the DeepSpeed library.

DeepSpeed version of NVIDIA's Megatron-LM that adds additional support for several features such as MoE model training, Curriculum Learning, 3D Parallelism, and others.

A Native-PyTorch Library for LLM Fine-tuning.

Efficient Training for Big Models.

Mesh TensorFlow: Model Parallelism Made Easier.

A library for accelerating Transformer model training on NVIDIA GPUs.