exllama
Inference Engines
exllama

A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights.

GitHub
A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights.

Relevant Sites

Embedchain 26,094

Framework to create ChatGPT like bots over your dataset.

AI Gateway 7,293

Gateway streamlines requests to 100+ open & closed source models with a unified API. It is also production-ready with support for caching, fallbacks, retries, timeouts, loadbalancing, and can be edge-deployed for minimum latency.

LMDeploy 5,838

A high-throughput and low-latency inference and serving framework for LLMs and VLs

Tune Studio

Playground for devs to finetune & deploy LLMs

Sidekick

Data integration platform for LLMs.

TensorRT-LLM 9,701

Nvidia Framework for LLM Inference

Leave a Reply

Your email address will not be published. Required fields are marked *