promptfoo | LLMWay – The Way To LLM

Inference Engines

promptfoo

Test your prompts. Evaluate and compare LLM outputs, catch regressions, and improve prompt quality.

GitHub

Test your prompts. Evaluate and compare LLM outputs, catch regressions, and improve prompt quality.

DeepSpeed-Mii 2,034

MII makes low-latency and high-throughput inference, similar to vLLM powered by DeepSpeed.

llama.cpp 83,097

LLM inference in C/C++.

Text-Embeddings-Inference 3,801

Inference for text-embeddings in Rust, HFOIL Licence.

wechat-chatgpt 13,307

Use ChatGPT On Wechat via wechaty

talkd.ai dialog 403

Simple API for deploying any RAG or LLM that you want adding plugins.

Infinity 2,311

Inference for text-embeddings in Python