Introduction
What is Paddler? Why Paddler? Before you begin Installation Changelog
Starting out
Set up a basic LLM cluster Using the web admin panel Generating tokens and embeddings Inference parameters Using function calling Using grammars Using multimodal models Multi-agent fleet Going beyond a single device Running the balancer with state database Managing Paddler through API Reporting metrics Model swapping
Paddler in your project
For ops teams For product teams
Best practices
How to control response quality How to handle the load
Migrating to Paddler
OpenAI compatibility
Desktop app (beta feature)
Introduction Installation Using the desktop app
Internals
How Paddler works Buffered requests How to enable embeddings CORS configuration Chat templates

Installation

There are multiple ways to install Paddler.

The goal is to somehow obtain the paddler binary and make it available in your system.

Obtaining the Paddler binary

Option 1: Download the latest release

You can download the latest release from our GitHub releases.

Option 2: Build from source

If your system is not supported by the pre-built binaries (or you just want to build it yourself), you can build Paddler from source. Paddler current MSRV is 1.88.0.

To do that:

  1. Have Rust and Node installed on your system.

  2. Clone our repository.

  3. From the root directory of the project, run the build command for your hardware.

Hardware Build command Binary path
CPU make target/release/paddler
Cuda make target/cuda/release/paddler target/cuda/release/paddler
Metal make target/metal/release/paddler target/metal/release/paddler
Vulkan make target/vulkan/release/paddler target/vulkan/release/paddler

To build with Cuda, you also need to have the CUDA toolkit installed on your system, and the nvcc compiler must be in your PATH.

Need more? 🙂

Start an issue on GitHub.

Using Paddler

The entire Paddler functionality is available through the paddler command.

You can run paddler --help to see the available commands and options.

If you want to install it globally in your system, you can usually move the binary into /usr/local/bin, or to a similar directory that is in your PATH. That is system-specific, though, so it is not viable to cover all the possible options here. Look up "how to install a binary globally on Linux/Windows/Mac/BSD/*".

Before you begin Changelog