AI Gateway
Inference Engines
AI Gateway

Gateway streamlines requests to 100+ open & closed source models with a unified API. It is also production-ready with support for caching, fallbacks, retries, timeouts, loadbalancing, and can be edge-deployed for minimum latency.

GitHub
Gateway streamlines requests to 100+ open & closed source models with a unified API. It is also production-ready with support for caching, fallbacks, retries, timeouts, loadbalancing, and can be edge-deployed for minimum latency.

Relevant Sites

LMDeploy 7,034

A high-throughput and low-latency inference and serving framework for LLMs and VLs

exllama 2,897

A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights.

Agenta 3,151

Easily build, version, evaluate and deploy your LLM-powered apps.

ollama 152,237

Get up and running with Llama 3, Mistral, Gemma, and other large language models.

Swiss Army Llama 1,029

Comprehensive set of tools for working with local LLMs for various tasks.

FastChat 39,085

A distributed multi-model LLM serving system with web UI and OpenAI-compatible RESTful APIs.

Leave a Reply

Your email address will not be published. Required fields are marked *