Fine-tune, serve, deploy, and monitor any open-source LLMs in production. Used in production at BentoML for LLMs-based applications.

A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights.

simplifies the evaluation of LLMs by providing a unified microservice to access and test multiple AI models.

Simple API for deploying any RAG or LLM that you want adding plugins.

Confidently evaluate, test, and ship LLM applications with a suite of observability tools to calibrate language model outputs across your dev and production lifecycle.

Nvidia Framework for LLM Inference