NanoFlow is a throughput-oriented high-performance serving framework for LLMs. NanoFlow consistently delivers superior throughput compared to vLLM, Deepspeed-FastGen, and TensorRT-LLM.

a chat interface crafted with llama.cpp for running Alpaca models. No API keys, entirely self-hosted!

Seamlessly integrate LLMs as Python functions

a toolkit for deploying and serving Large Language Models (LLMs).

Easily build, version, evaluate and deploy your LLM-powered apps.

Create, deploy and operate Actions using Python anywhere to enhance your AI agents and assistants. Batteries included with an extensive set of libraries, helpers and logging.