Introduction
What is Paddler? Why Paddler? Before you begin Installation Changelog
Starting out
Set up a basic LLM cluster Using the web admin panel Generating tokens and embeddings Using function calling Multi-agent fleet Going beyond a single device Running the balancer with state database Managing Paddler through API Reporting metrics Model swapping
Paddler in your project
For ops teams For product teams
Best practices
How to control response quality How to handle the load
Migrating to Paddler
OpenAI compatibility
Internals
How Paddler works Buffered requests How to enable embeddings CORS configuration Chat templates

Installation

There are multiple ways to install Paddler.

The goal is to somehow obtain the paddler binary and make it available in your system.

Obtaining the Paddler binary

Option 1: Download the latest release

You can download the latest release from our GitHub releases.

Option 2: Build from source

If your system is not supported by the pre-built binaries (or you just want to build it yourself), you can build Paddler from source.

To do that, follow the steps:

  1. Have Rust and Node installed on your system.

  2. Clone our repository and run make release in the root directory of the project.

  3. That will produce a paddler binary in the target/release directory.

Hardware support

Cuda

You need to compile Paddler with the cuda feature (cargo build --features cuda).

You need to have the CUDA toolkit installed on your system, and the nvcc compiler must be in your PATH.

Metal

Works out of the box on Macs.

Vulkan

You need to compile Paddler with the vulkan feature (cargo build --features vulkan).

Need more? 🙂

Start an issue on GitHub.

Using Paddler

The entire Paddler functionality is available through the paddler command.

You can run paddler --help to see the available commands and options.

If you want to install it globally in your system, you can usually move the binary into /usr/local/bin, or to a similar directory that is in your PATH. That is system-specific, though, so it is not viable to cover all the possible options here. Look up "how to install a binary globally on Linux/Windows/Mac/BSD/*".

Before you begin Changelog