FasterTransformer | LLMWay

Inference Engines

FasterTransformer

NVIDIA Framework for LLM Inference(Transitioned to TensorRT-LLM)

GitHub

NVIDIA Framework for LLM Inference(Transitioned to TensorRT-LLM)

wechat-chatgpt 13,340

Use ChatGPT On Wechat via wechaty

FastChat 38,108

A distributed multi-model LLM serving system with web UI and OpenAI-compatible RESTful APIs.

promptfoo 5,826

Test your prompts. Evaluate and compare LLM outputs, catch regressions, and improve prompt quality.

TensorRT-LLM 9,701

Nvidia Framework for LLM Inference

exllama 2,840

A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights.

Langfuse 9,357

Open Source LLM Engineering Platform 🪢 Tracing, Evaluations, Prompt Management, Evaluations and Playground.