Introduction
Using Paddler API Streaming tokens
Inference service
Continue from conversation history Continue from raw prompt Generate embedding batch WebSocket endpoint
Management service
Get agents Get agents stream Get balancer desired state Get buffered requests Get buffered requests stream Get chat template override Get model metadata Put balancer desired state Get metrics
Compatibility Services
OpenAI

Using Paddler API

Paddler is divided into two services: the Inference service and the Management service. Each service has its own API and uses a separate address.

The Inference service provides tokens and embeddings. You can send requests to the Inference service through either WebSocket or REST.

The Management service manages the whole Paddler setup (get the list of registered agents, model metadata, specify the balancer's desired state, etc.). It is managed through REST.

Although the Management service does offer a WebSocket connection, it is reserved for agents to maintain a persistent connection with the balancer. We don't document it here since it's intended only for internal use.

Streaming tokens