MII makes low-latency and high-throughput inference, similar to vLLM powered by DeepSpeed.

a toolkit for deploying and serving Large Language Models (LLMs).

Locally running websearch using LLM chains

An open-source GPU cluster manager for running LLMs

AI gateway and marketplace for developers, enables streamlined integration of AI features into products

SGLang is a fast serving framework for large language models and vision language models.