a toolkit for deploying and serving Large Language Models (LLMs).

SGLang is a fast serving framework for large language models and vision language models.

FlexLLMGen is a high-throughput generation engine for running large language models with limited GPU memory. FlexLLMGen allows high-throughput generation by IO-efficient offloading, compression, and large effective batch sizes.

Build your own conversational search engine using less than 500 lines of code by LeptonAI.

an open-source NLP framework that allows you to use LLMs and transformer-based models from Hugging Face, OpenAI and Cohere to interact with your own data.

Comprehensive set of tools for working with local LLMs for various tasks.