MNN-LLM | LLMWay – The Way To LLM

Inference Engines

MNN-LLM

A Device-Inference framework, including LLM Inference on device(Mobile Phone/PC/IOT)

GitHub

A Device-Inference framework, including LLM Inference on device(Mobile Phone/PC/IOT)

TensorRT-LLM 12,074

Nvidia Framework for LLM Inference

LiteChain 421

Lightweight alternative to LangChain for composing LLMs

Serge 5,754

a chat interface crafted with llama.cpp for running Alpaca models. No API keys, entirely self-hosted!

Wllama 930

WebAssembly binding for llama.cpp - Enabling in-browser LLM inference

FastChat 39,241

A distributed multi-model LLM serving system with web UI and OpenAI-compatible RESTful APIs.

LLocalSearch 5,951

Locally running websearch using LLM chains