LMDeploy | LLMWay – The Way To LLM

Inference Engines

LMDeploy

A high-throughput and low-latency inference and serving framework for LLMs and VLs

GitHub

A high-throughput and low-latency inference and serving framework for LLMs and VLs

exllama 2,840

A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights.

llama.cpp 76,497

LLM inference in C/C++.

Floom 41

AI gateway and marketplace for developers, enables streamlined integration of AI features into products

Embedchain 26,094

Framework to create ChatGPT like bots over your dataset.