OpenLLM
Inference Engines
OpenLLM

Fine-tune, serve, deploy, and monitor any open-source LLMs in production. Used in production at BentoML for LLMs-based applications.

GitHub
Fine-tune, serve, deploy, and monitor any open-source LLMs in production. Used in production at BentoML for LLMs-based applications.

Relevant Sites

TensorRT-LLM 9,701

Nvidia Framework for LLM Inference

promptfoo 5,826

Test your prompts. Evaluate and compare LLM outputs, catch regressions, and improve prompt quality.

AI Gateway 7,293

Gateway streamlines requests to 100+ open & closed source models with a unified API. It is also production-ready with support for caching, fallbacks, retries, timeouts, loadbalancing, and can be edge-deployed for minimum latency.

FastChat 38,108

A distributed multi-model LLM serving system with web UI and OpenAI-compatible RESTful APIs.

MNN-LLM 10,008

A Device-Inference framework, including LLM Inference on device(Mobile Phone/PC/IOT)

SGLang 11,910

SGLang is a fast serving framework for large language models and vision language models.

Leave a Reply

Your email address will not be published. Required fields are marked *