A Device-Inference framework, including LLM Inference on device(Mobile Phone/PC/IOT)

Framework to create ChatGPT like bots over your dataset.

Open Source LLM Engineering Platform 🪢 Tracing, Evaluations, Prompt Management, Evaluations and Playground.

NanoFlow is a throughput-oriented high-performance serving framework for LLMs. NanoFlow consistently delivers superior throughput compared to vLLM, Deepspeed-FastGen, and TensorRT-LLM.

Simple API for deploying any RAG or LLM that you want adding plugins.

Lightweight alternative to LangChain for composing LLMs