NVIDIA Framework for LLM Inference(Transitioned to TensorRT-LLM)

Get up and running with Llama 3, Mistral, Gemma, and other large language models.

A method designed to enhance the efficiency of Transformer models

Confidently evaluate, test, and ship LLM applications with a suite of observability tools to calibrate language model outputs across your dev and production lifecycle.

MII makes low-latency and high-throughput inference, similar to vLLM powered by DeepSpeed.

FlexLLMGen is a high-throughput generation engine for running large language models with limited GPU memory. FlexLLMGen allows high-throughput generation by IO-efficient offloading, compression, and large effective batch sizes.