SGLang is a fast serving framework for large language models and vision language models.

A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights.

Confidently evaluate, test, and ship LLM applications with a suite of observability tools to calibrate language model outputs across your dev and production lifecycle.

Simple API for deploying any RAG or LLM that you want adding plugins.

simplifies the evaluation of LLMs by providing a unified microservice to access and test multiple AI models.

Fine-tune, serve, deploy, and monitor any open-source LLMs in production. Used in production at BentoML for LLMs-based applications.