Simple API for deploying any RAG or LLM that you want adding plugins.

Building applications with LLMs through composability

Gateway streamlines requests to 100+ open & closed source models with a unified API. It is also production-ready with support for caching, fallbacks, retries, timeouts, loadbalancing, and can be edge-deployed for minimum latency.

Fine-tune, serve, deploy, and monitor any open-source LLMs in production. Used in production at BentoML for LLMs-based applications.

SGLang is a fast serving framework for large language models and vision language models.

a chat interface crafted with llama.cpp for running Alpaca models. No API keys, entirely self-hosted!