SGLang
Inference Engines
SGLang

SGLang is a fast serving framework for large language models and vision language models.

GitHub
SGLang is a fast serving framework for large language models and vision language models.

Relevant Sites

mistral.rs 6,211

Blazingly fast LLM inference.

GPUStack 3,973

An open-source GPU cluster manager for running LLMs

Embedchain 42,828

Framework to create ChatGPT like bots over your dataset.

AI Gateway 9,819

Gateway streamlines requests to 100+ open & closed source models with a unified API. It is also production-ready with support for caching, fallbacks, retries, timeouts, loadbalancing, and can be edge-deployed for minimum latency.

magentic 2,378

Seamlessly integrate LLMs as Python functions

vLLM 62,534

A high-throughput and memory-efficient inference and serving engine for LLMs.

Leave a Reply

Your email address will not be published. Required fields are marked *