NanoFlow is a throughput-oriented high-performance serving framework for LLMs. NanoFlow consistently delivers superior throughput compared to vLLM, Deepspeed-FastGen, and TensorRT-LLM.

Seamlessly integrate LLMs as Python functions

Comprehensive set of tools for working with local LLMs for various tasks.

An interactive chat project that leverages Ollama/OpenAI/MistralAI LLMs for rapid understanding and navigation of GitHub code repository or compressed file resources.

Test your prompts. Evaluate and compare LLM outputs, catch regressions, and improve prompt quality.

WebAssembly binding for llama.cpp - Enabling in-browser LLM inference