Get up and running with Llama 3, Mistral, Gemma, and other large language models.

Lightweight alternative to LangChain for composing LLMs

FlexLLMGen is a high-throughput generation engine for running large language models with limited GPU memory. FlexLLMGen allows high-throughput generation by IO-efficient offloading, compression, and large effective batch sizes.

A method designed to enhance the efficiency of Transformer models

NVIDIA Framework for LLM Inference(Transitioned to TensorRT-LLM)

A high-throughput and memory-efficient inference and serving engine for LLMs.