FlexLLMGen is a high-throughput generation engine for running large language models with limited GPU memory. FlexLLMGen allows high-throughput generation by IO-efficient offloading, compression, and large effective batch sizes.

Build your own conversational search engine using less than 500 lines of code by LeptonAI.

Nvidia Framework for LLM Inference

Test your prompts. Evaluate and compare LLM outputs, catch regressions, and improve prompt quality.

Locally running websearch using LLM chains

MII makes low-latency and high-throughput inference, similar to vLLM powered by DeepSpeed.