Simple API for deploying any RAG or LLM that you want adding plugins.

A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights.

First LLM Multi-agent framework.

NanoFlow is a throughput-oriented high-performance serving framework for LLMs. NanoFlow consistently delivers superior throughput compared to vLLM, Deepspeed-FastGen, and TensorRT-LLM.

LLM inference in C/C++.

A method designed to enhance the efficiency of Transformer models