Nvidia Framework for LLM Inference

a chat interface crafted with llama.cpp for running Alpaca models. No API keys, entirely self-hosted!

LLM inference in C/C++.

A Device-Inference framework, including LLM Inference on device(Mobile Phone/PC/IOT)

NanoFlow is a throughput-oriented high-performance serving framework for LLMs. NanoFlow consistently delivers superior throughput compared to vLLM, Deepspeed-FastGen, and TensorRT-LLM.

Comprehensive set of tools for working with local LLMs for various tasks.