Building applications with LLMs through composability

Locally running websearch using LLM chains

Gateway streamlines requests to 100+ open & closed source models with a unified API. It is also production-ready with support for caching, fallbacks, retries, timeouts, loadbalancing, and can be edge-deployed for minimum latency.

Get up and running with Llama 3, Mistral, Gemma, and other large language models.

Inference for text-embeddings in Python

A high-throughput and low-latency inference and serving framework for LLMs and VLs