Get up and running with Llama 3, Mistral, Gemma, and other large language models.

a chat interface crafted with llama.cpp for running Alpaca models. No API keys, entirely self-hosted!

LLM inference in C/C++.

Confidently evaluate, test, and ship LLM applications with a suite of observability tools to calibrate language model outputs across your dev and production lifecycle.

Framework to create ChatGPT like bots over your dataset.

MII makes low-latency and high-throughput inference, similar to vLLM powered by DeepSpeed.