Use ChatGPT On Wechat via wechaty

Data integration platform for LLMs.

NVIDIA Framework for LLM Inference(Transitioned to TensorRT-LLM)

A high-throughput and memory-efficient inference and serving engine for LLMs.

Nvidia Framework for LLM Inference

FlexLLMGen is a high-throughput generation engine for running large language models with limited GPU memory. FlexLLMGen allows high-throughput generation by IO-efficient offloading, compression, and large effective batch sizes.