A Device-Inference framework, including LLM Inference on device(Mobile Phone/PC/IOT)

Inference for text-embeddings in Python

Fine-tune, serve, deploy, and monitor any open-source LLMs in production. Used in production at BentoML for LLMs-based applications.

A high-throughput and low-latency inference and serving framework for LLMs and VLs

First LLM Multi-agent framework.

Confidently evaluate, test, and ship LLM applications with a suite of observability tools to calibrate language model outputs across your dev and production lifecycle.