Fine-tune, serve, deploy, and monitor any open-source LLMs in production. Used in production at BentoML for LLMs-based applications.

A Device-Inference framework, including LLM Inference on device(Mobile Phone/PC/IOT)

An open-source GPU cluster manager for running LLMs

SGLang is a fast serving framework for large language models and vision language models.

Comprehensive set of tools for working with local LLMs for various tasks.

Playground for devs to finetune & deploy LLMs