An implementation of model parallel autoregressive transformers on GPUs, based on the DeepSpeed library.

Ongoing research training transformer models at scale.

Efficient Training for Big Models.

Mesh TensorFlow: Model Parallelism Made Easier.

Generative AI framework built for researchers and PyTorch developers working on Large Language Models (LLMs), Multimodal Models (MMs), Automatic Speech Recognition (ASR), Text to Speech (TTS), and Computer Vision (CV) domains.

A Native-PyTorch Library for LLM Fine-tuning.