SkyPilot | LLMWay – The Way To LLM

Inference Engines

SkyPilot

Run LLMs and batch jobs on any cloud. Get maximum cost savings, highest GPU availability, and managed execution -- all with a simple interface.

GitHub

Run LLMs and batch jobs on any cloud. Get maximum cost savings, highest GPU availability, and managed execution -- all with a simple interface.

llama.cpp 80,245

LLM inference in C/C++.

Floom 42

AI gateway and marketplace for developers, enables streamlined integration of AI features into products

DeepSpeed-Mii 2,009

MII makes low-latency and high-throughput inference, similar to vLLM powered by DeepSpeed.

Serge 5,717

a chat interface crafted with llama.cpp for running Alpaca models. No API keys, entirely self-hosted!

LiteChain 418

Lightweight alternative to LangChain for composing LLMs