MII makes low-latency and high-throughput inference, similar to vLLM powered by DeepSpeed.

an open-source NLP framework that allows you to use LLMs and transformer-based models from Hugging Face, OpenAI and Cohere to interact with your own data.

A high-throughput and low-latency inference and serving framework for LLMs and VLs

Formerly langchain-ChatGLM, local knowledge based LLM (like ChatGLM) QA app with langchain.

Harness LLMs with Multi-Agent Programming

FlexLLMGen is a high-throughput generation engine for running large language models with limited GPU memory. FlexLLMGen allows high-throughput generation by IO-efficient offloading, compression, and large effective batch sizes.