A Device-Inference framework, including LLM Inference on device(Mobile Phone/PC/IOT)

Interact with LLM using Ollama models(or openAI, mistralAI)via pure shell scripts on your Linux(or MacOS) system, enhancing intelligent system management without any dependencies.

A high-throughput and low-latency inference and serving framework for LLMs and VLs

Blazingly fast LLM inference.

Easily build, version, evaluate and deploy your LLM-powered apps.

A high-throughput and memory-efficient inference and serving engine for LLMs.