Haystack
Inference Engines
Haystack

an open-source NLP framework that allows you to use LLMs and transformer-based models from Hugging Face, OpenAI and Cohere to interact with your own data.

Link
an open-source NLP framework that allows you to use LLMs and transformer-based models from Hugging Face, OpenAI and Cohere to interact with your own data.

Relevant Sites

talkd.ai dialog 403

Simple API for deploying any RAG or LLM that you want adding plugins.

LMDeploy 6,714

A high-throughput and low-latency inference and serving framework for LLMs and VLs

Opik 11,327

Confidently evaluate, test, and ship LLM applications with a suite of observability tools to calibrate language model outputs across your dev and production lifecycle.

DeepSpeed-Mii 2,034

MII makes low-latency and high-throughput inference, similar to vLLM powered by DeepSpeed.

mistral.rs 5,890

Blazingly fast LLM inference.

LiteChain 419

Lightweight alternative to LangChain for composing LLMs

Leave a Reply

Your email address will not be published. Required fields are marked *