a toolkit for deploying and serving Large Language Models (LLMs).

Run LLMs and batch jobs on any cloud. Get maximum cost savings, highest GPU availability, and managed execution -- all with a simple interface.

A Device-Inference framework, including LLM Inference on device(Mobile Phone/PC/IOT)

Comprehensive set of tools for working with local LLMs for various tasks.

Simple API for deploying any RAG or LLM that you want adding plugins.

WebAssembly binding for llama.cpp - Enabling in-browser LLM inference