LLM inference in C/C++.

Nvidia Framework for LLM Inference

NVIDIA Framework for LLM Inference(Transitioned to TensorRT-LLM)

Gateway streamlines requests to 100+ open & closed source models with a unified API. It is also production-ready with support for caching, fallbacks, retries, timeouts, loadbalancing, and can be edge-deployed for minimum latency.

Lightweight alternative to LangChain for composing LLMs

NanoFlow is a throughput-oriented high-performance serving framework for LLMs. NanoFlow consistently delivers superior throughput compared to vLLM, Deepspeed-FastGen, and TensorRT-LLM.