FlexLLMGen is a high-throughput generation engine for running large language models with limited GPU memory. FlexLLMGen allows high-throughput generation by IO-efficient offloading, compression, and large effective batch sizes.

An open-source GPU cluster manager for running LLMs

Lightweight alternative to LangChain for composing LLMs

Seamlessly integrate LLMs as Python functions

SGLang is a fast serving framework for large language models and vision language models.

Use ChatGPT On Wechat via wechaty