a toolkit for deploying and serving Large Language Models (LLMs).

Blazingly fast LLM inference.

Interact with LLM using Ollama models(or openAI, mistralAI)via pure shell scripts on your Linux(or MacOS) system, enhancing intelligent system management without any dependencies.

First LLM Multi-agent framework.

Gateway streamlines requests to 100+ open & closed source models with a unified API. It is also production-ready with support for caching, fallbacks, retries, timeouts, loadbalancing, and can be edge-deployed for minimum latency.

Simple API for deploying any RAG or LLM that you want adding plugins.