Simple API for deploying any RAG or LLM that you want adding plugins.

MII makes low-latency and high-throughput inference, similar to vLLM powered by DeepSpeed.

A high-throughput and memory-efficient inference and serving engine for LLMs.

LLM inference in C/C++.

Interact with LLM using Ollama models(or openAI, mistralAI)via pure shell scripts on your Linux(or MacOS) system, enhancing intelligent system management without any dependencies.

NVIDIA Framework for LLM Inference(Transitioned to TensorRT-LLM)