A distributed multi-model LLM serving system with web UI and OpenAI-compatible RESTful APIs.

A Device-Inference framework, including LLM Inference on device(Mobile Phone/PC/IOT)

A high-throughput and low-latency inference and serving framework for LLMs and VLs

Framework to create ChatGPT like bots over your dataset.

Gateway streamlines requests to 100+ open & closed source models with a unified API. It is also production-ready with support for caching, fallbacks, retries, timeouts, loadbalancing, and can be edge-deployed for minimum latency.

MII makes low-latency and high-throughput inference, similar to vLLM powered by DeepSpeed.