A method designed to enhance the efficiency of Transformer models

A Device-Inference framework, including LLM Inference on device(Mobile Phone/PC/IOT)

FlexLLMGen is a high-throughput generation engine for running large language models with limited GPU memory. FlexLLMGen allows high-throughput generation by IO-efficient offloading, compression, and large effective batch sizes.

WebAssembly binding for llama.cpp - Enabling in-browser LLM inference

A high-throughput and low-latency inference and serving framework for LLMs and VLs

NanoFlow is a throughput-oriented high-performance serving framework for LLMs. NanoFlow consistently delivers superior throughput compared to vLLM, Deepspeed-FastGen, and TensorRT-LLM.