simplifies the evaluation of LLMs by providing a unified microservice to access and test multiple AI models.

FlexLLMGen is a high-throughput generation engine for running large language models with limited GPU memory. FlexLLMGen allows high-throughput generation by IO-efficient offloading, compression, and large effective batch sizes.

Inference for text-embeddings in Python

a chat interface crafted with llama.cpp for running Alpaca models. No API keys, entirely self-hosted!

Interact with LLM using Ollama models(or openAI, mistralAI)via pure shell scripts on your Linux(or MacOS) system, enhancing intelligent system management without any dependencies.

Fine-tune, serve, deploy, and monitor any open-source LLMs in production. Used in production at BentoML for LLMs-based applications.