A high-throughput and memory-efficient inference and serving engine for LLMs.
SGLang is a fast serving framework for large language models and vision language models.
a toolkit for deploying and serving Large Language Models (LLMs).
A high-throughput and low-latency inference and serving framework for LLMs and VLs
Get up and running with Llama 3, Mistral, Gemma, and other large language models.
NanoFlow is a throughput-oriented high-performance serving framework for LLMs. NanoFlow consistently delivers superior throughput compared to vLLM, Deepspeed-FastGen, and TensorRT-LLM.
LLM inference in C/C++.
Open Source LLM Engineering Platform 🪢 Tracing, Evaluations, Prompt Management, Evaluations and Playground.
A distributed multi-model LLM serving system with web UI and OpenAI-compatible RESTful APIs.
Blazingly fast LLM inference.
A python package for Txt-to-SQL with self hosting functionalities and RESTful APIs compatible with proprietary as well as open source LLM.
Run LLMs and batch jobs on any cloud. Get maximum cost savings, highest GPU availability, and managed execution -- all with a simple interface.
an open-source NLP framework that allows you to use LLMs and transformer-based models from Hugging Face, OpenAI and Cohere to interact with your own data.
Data integration platform for LLMs.
A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights.
An interactive chat project that leverages Ollama/OpenAI/MistralAI LLMs for rapid understanding and navigation of GitHub code repository or compressed file resources.
Interact with LLM using Ollama models(or openAI, mistralAI)via pure shell scripts on your Linux(or MacOS) system, enhancing intelligent system management without any dependencies.
Building applications with LLMs through composability
AI gateway and marketplace for developers, enables streamlined integration of AI features into products
Comprehensive set of tools for working with local LLMs for various tasks.
Lightweight alternative to LangChain for composing LLMs
Seamlessly integrate LLMs as Python functions
Use ChatGPT On Wechat via wechaty
Test your prompts. Evaluate and compare LLM outputs, catch regressions, and improve prompt quality.
Easily build, version, evaluate and deploy your LLM-powered apps.
a chat interface crafted with llama.cpp for running Alpaca models. No API keys, entirely self-hosted!
Harness LLMs with Multi-Agent Programming
Framework to create ChatGPT like bots over your dataset.
Confidently evaluate, test, and ship LLM applications with a suite of observability tools to calibrate language model outputs across your dev and production lifecycle.
simplifies the evaluation of LLMs by providing a unified microservice to access and test multiple AI models.
Fine-tune, serve, deploy, and monitor any open-source LLMs in production. Used in production at BentoML for LLMs-based applications.
MII makes low-latency and high-throughput inference, similar to vLLM powered by DeepSpeed.
Inference for text-embeddings in Rust, HFOIL Licence.
Inference for text-embeddings in Python
Nvidia Framework for LLM Inference
NVIDIA Framework for LLM Inference(Transitioned to TensorRT-LLM)
A method designed to enhance the efficiency of Transformer models
Formerly langchain-ChatGLM, local knowledge based LLM (like ChatGLM) QA app with langchain.
Build your own conversational search engine using less than 500 lines of code by LeptonAI.
Create, deploy and operate Actions using Python anywhere to enhance your AI agents and assistants. Batteries included with an extensive set of libraries, helpers and logging.
Playground for devs to finetune & deploy LLMs
Locally running websearch using LLM chains
Gateway streamlines requests to 100+ open & closed source models with a unified API. It is also production-ready with support for caching, fallbacks, retries, timeouts, loadbalancing, and can be edge-deployed for minimum latency.
Simple API for deploying any RAG or LLM that you want adding plugins.
WebAssembly binding for llama.cpp - Enabling in-browser LLM inference
An open-source GPU cluster manager for running LLMs
A Device-Inference framework, including LLM Inference on device(Mobile Phone/PC/IOT)
First LLM Multi-agent framework.
FlexLLMGen is a high-throughput generation engine for running large language models with limited GPU memory. FlexLLMGen allows high-throughput generation by IO-efficient offloading, compression, and large effective batch sizes.
A high-throughput and memory-efficient inference and serving engine for LLMs.