MII makes low-latency and high-throughput inference, similar to vLLM powered by DeepSpeed.

Harness LLMs with Multi-Agent Programming

NVIDIA Framework for LLM Inference(Transitioned to TensorRT-LLM)

NanoFlow is a throughput-oriented high-performance serving framework for LLMs. NanoFlow consistently delivers superior throughput compared to vLLM, Deepspeed-FastGen, and TensorRT-LLM.

Blazingly fast LLM inference.

Gateway streamlines requests to 100+ open & closed source models with a unified API. It is also production-ready with support for caching, fallbacks, retries, timeouts, loadbalancing, and can be edge-deployed for minimum latency.