First LLM Multi-agent framework.

Nvidia Framework for LLM Inference

A distributed multi-model LLM serving system with web UI and OpenAI-compatible RESTful APIs.

MII makes low-latency and high-throughput inference, similar to vLLM powered by DeepSpeed.

NVIDIA Framework for LLM Inference(Transitioned to TensorRT-LLM)

Create, deploy and operate Actions using Python anywhere to enhance your AI agents and assistants. Batteries included with an extensive set of libraries, helpers and logging.