Building applications with LLMs through composability

Fine-tune, serve, deploy, and monitor any open-source LLMs in production. Used in production at BentoML for LLMs-based applications.

NVIDIA Framework for LLM Inference(Transitioned to TensorRT-LLM)

Comprehensive set of tools for working with local LLMs for various tasks.

A distributed multi-model LLM serving system with web UI and OpenAI-compatible RESTful APIs.

Test your prompts. Evaluate and compare LLM outputs, catch regressions, and improve prompt quality.