An open-source GPU cluster manager for running LLMs

Test your prompts. Evaluate and compare LLM outputs, catch regressions, and improve prompt quality.

NanoFlow is a throughput-oriented high-performance serving framework for LLMs. NanoFlow consistently delivers superior throughput compared to vLLM, Deepspeed-FastGen, and TensorRT-LLM.

a toolkit for deploying and serving Large Language Models (LLMs).

Inference for text-embeddings in Rust, HFOIL Licence.

NVIDIA Framework for LLM Inference(Transitioned to TensorRT-LLM)