NanoFlow is a throughput-oriented high-performance serving framework for LLMs. NanoFlow consistently delivers superior throughput compared to vLLM, Deepspeed-FastGen, and TensorRT-LLM.

Building applications with LLMs through composability

MII makes low-latency and high-throughput inference, similar to vLLM powered by DeepSpeed.

WebAssembly binding for llama.cpp - Enabling in-browser LLM inference

Locally running websearch using LLM chains

Create, deploy and operate Actions using Python anywhere to enhance your AI agents and assistants. Batteries included with an extensive set of libraries, helpers and logging.