SGLang is a fast serving framework for large language models and vision language models.

Data integration platform for LLMs.

LLM inference in C/C++.

Seamlessly integrate LLMs as Python functions

NanoFlow is a throughput-oriented high-performance serving framework for LLMs. NanoFlow consistently delivers superior throughput compared to vLLM, Deepspeed-FastGen, and TensorRT-LLM.

Locally running websearch using LLM chains