Ongoing research training transformer models at scale.

Mesh TensorFlow: Model Parallelism Made Easier.

An implementation of model parallel autoregressive transformers on GPUs, based on the DeepSpeed library.

Making large AI models cheaper, faster, and more accessible.

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

A native PyTorch Library for large model training.