vLLM

Overview

vLLM is a high-throughput inference server for open-weight large language models. This template serves one Hugging Face model through vLLM’s OpenAI-compatible API (/v1/chat/completions, /v1/completions, /v1/models) on an NVIDIA GPU: either one of Control Plane’s built-in A10G or T4 cards, or your own GPU cluster registered as a BYOK location. The API is protected by an API key you create before installing, and the downloaded model is cached on a persistent volume so restarts do not download it again.

This template deploys into an existing GVC that you already have. It does not create, provision, or manage a GVC. Use a GVC with exactly one location: every location of the GVC runs its own copy of the server, each with its own GPUs.

What Gets Created

Resource	Name	Purpose	Optional
Workload	`RELEASE_NAME-vllm`	Stateful vLLM server, container `vllm` on port `8000`, one GPU set per replica (`replicas: 1` by default); `/metrics` is scraped into the platform’s built-in metrics	—
Volume Set	`RELEASE_NAME-vllm-models`	One `ext4` volume per replica at `/models`, holding the downloaded weights and vLLM’s compile cache; a re-downloadable cache, so no snapshots are kept	—
Identity	`RELEASE_NAME-vllm-identity`	Identity the workload runs as	—
Policy	`RELEASE_NAME-vllm-policy`	`reveal` on exactly your API key secret, plus your Hugging Face token secret when set	—

vLLM has no browser UI. For a chat UI, pair it with the Open WebUI template (see Pairing with Open WebUI).

Prerequisites

Create the API key secret

Clients send this key as a Bearer token. It is an opaque secret and must exist before you install:

printf '%s' "$(openssl rand -hex 32)" | cpln secret create-opaque --name API_KEY_SECRET_NAME --encoding plain -f -

Set apiKey.secretName to this name.

Make a GPU available

Built-in GPU (gpu.type: builtin, the default). Your org needs GPU quota in a location that offers it — by default one NVIDIA A10G with 6 vCPU and 16 GiB of memory. Ask Control Plane support to enable it. A T4 is the other built-in choice (gpu.model: t4).Your own GPU cluster (gpu.type: byok). Any Kubernetes cluster registered as a Control Plane BYOK location works: an MK8s cluster with the BYOK add-on enabled, or a cluster you already run (EKS, GKE, AKS, on-premises). A cluster that is not registered as a location cannot run this template. See GPU Cluster Setup (BYOK) for what each cluster type needs.

Optional - create a Hugging Face token secret for gated models

Gated models such as Llama or Gemma need a Hugging Face token. Accept the model’s license on its huggingface.co page with the account that owns the token, and wait until the page shows access granted (some, like Llama, are reviewed by hand). Create a read token (a fine-grained token needs read access to public gated repos), then store it:

printf '%s' "$HF_TOKEN" | cpln secret create-opaque --name HF_TOKEN_SECRET_NAME --encoding plain -f -

Set huggingFace.tokenSecretName to this name. Without license access the replica restarts with exit code 1 (see Troubleshooting).

If a secret named in apiKey.secretName or huggingFace.tokenSecretName does not exist, the deployment waits silently and cpln logs shows nothing. Read status.versions[].message, which names the missing secret:

cpln workload get-deployments RELEASE_NAME-vllm --gvc GVC_NAME -o yaml

Once the secret exists the deployment recovers by itself, which can take up to about 10 minutes; keep polling.

GPU Cluster Setup (BYOK)

Every cluster used with gpu.type: byok needs:

NVIDIA GPU nodes with driver 580 or newer (the image is a CUDA 13 build) and a device plugin that exposes nvidia.com/gpu.
A RuntimeClass named nvidia. The template always requests it, and its handler differs per cluster type (see below). Without it the install fails with The GPU runtime class nvidia is not supported in this cluster. If you add it after installing BYOK, restart BYOK’s actuator, which does not see new runtime classes until then: kubectl rollout restart deploy/actuator -n controlplane.
No taint on the GPU nodes. Control Plane workloads do not tolerate it.
The GPU node label cpln.io/gpuUsageAutoDrain: ignore. Otherwise BYOK drains a GPU node that has had no GPU pod for 5 minutes, including one where vLLM is still starting.
A GPU node boot disk of at least 200 GB. The vLLM image and the model otherwise fill the disk and the node evicts vLLM.
resources sized to the node. A 4-vCPU GPU node (g5.xlarge, g6.xlarge, g2-standard-4, Standard_NC4as_T4_v3) cannot fit the built-in default of 6 CPU and the replica stays pending; use about cpu: 2000m and memory: 8Gi.
A GVC whose only location is the BYOK location. On a Control Plane location the GPU request is ignored and vLLM cannot start.

On EKS, GKE and AKS, create the RuntimeClass yourself (MK8s ships one):

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: nvidia
handler: nvidia # EKS: nvidia, GKE: runc, AKS: nvidia-container-runtime

Each of the following was tested with this template; the notes are what that cluster type needs beyond the list above.

MK8s on AWS

NVIDIA add-on with taintGPUNodes: false.
An x86 NVIDIA-driver node image (an AWS Deep Learning AMI) plus the label mk8s.cpln.io/accelerator: nvidia. The recommended +nvidia images currently resolve to arm64 and cannot boot GPU instances, and with a custom image the device plugin only runs on nodes carrying that label.
One instance type per GPU node pool.
The nvidia RuntimeClass is built in.

MK8s on GCP

NVIDIA add-on with taintGPUNodes: false.
A driver image on the GPU pool: overrideImage.family with project deeplearning-platform-release and family common-cu129-ubuntu-2404-nvidia-580, plus the label mk8s.cpln.io/accelerator: nvidia.
Only machine types with a built-in GPU work (G2 with L4, A2, A3). GCP node pools have no accelerator field, so a T4 or V100 on an N1 machine is not possible.
The nvidia RuntimeClass and the cloud load balancer work with no extra setup.

EKS

The Amazon Linux 2023 NVIDIA AMI for the GPU node group; eksctl selects it for GPU instance types and installs the device plugin.
RuntimeClass nvidia with handler nvidia.
The AWS Load Balancer Controller. BYOK’s ingress requests an external network load balancer and gets no address without it. Installing its Helm chart after BYOK needs --disable-openapi-validation.

GKE

A GPU node pool with gpu-driver-version=latest, so GKE installs the driver.
RuntimeClass nvidia with handler runc. GKE has no nvidia handler; it injects the driver itself.
On N1 machines (the only T4 family), replace BYOK’s storage classes. They create Hyperdisk volumes, which N1 machines cannot attach. Set byok.noDefaultStorageClasses: true in cluster.json of the ConfigMap cpln-byok-current in kube-system, restart deploy/cpln-byok-agent -n kube-system, and create your own general-purpose-ssd-ext4 storage class on pd-balanced. Machines with a built-in GPU (G2) attach Hyperdisk without this.
If the BYOK install stalls with istiod never created, restart deploy/cpln-byok-agent -n kube-system.

AKS

A GPU node pool in an availability zone (for example --zones 1). BYOK’s storage classes use Premium SSD v2, which Azure attaches only to zonal VMs; on a non-zonal pool the volume fails with can be used only with Virtual Machines in an Availability Zone.
The default --gpu-driver Install, which installs driver 580.
The NVIDIA device plugin. AKS does not install it. Give the GPU pool the label nvidia.com/gpu.present=true so the plugin’s Helm chart schedules there.
RuntimeClass nvidia with handler nvidia-container-runtime.

MK8s on Azure does not run this template yet: a GPU node pool with a custom NVIDIA-driver image is not created, and the default image has no driver.

Installation

Install with the name of your API key secret:

cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/vllm \
  --version 1.0.0 --gvc GVC_NAME \
  --set apiKey.secretName=API_KEY_SECRET_NAME

The first start pulls a large image and then downloads and loads the model, so allow several minutes before the API answers.

UI

Browse, install, and manage templates visually

CLI

Manage templates from your terminal

Terraform

Declare templates in your Terraform configurations

Pulumi

Declare templates in your Pulumi programs

Configuration

Image

image: vllm/vllm-openai:v0.31.0 # CUDA 13 build; needs NVIDIA driver 580 or newer (Control Plane GPUs run 580)

Model

The model is downloaded into the volume on first start. Any extra vllm serve flag goes in extraArgs (see the engine arguments).

model:
  name: Qwen/Qwen2.5-1.5B-Instruct # Hugging Face model id; downloaded into the volume on first start
  servedModelName: "" # name clients send as "model"; empty = the model id
  maxModelLen: 8192 # max tokens per request (prompt + output)
  gpuMemoryUtilization: 0.90 # share of GPU memory vLLM may use for weights and KV cache
  dtype: auto # options: auto, half, bfloat16, float32 (bfloat16 is not supported on t4)
  toolCallParser: hermes # tool calling, which Open WebUI and agents need; must match the model: hermes (Qwen), llama3_json (Llama 3), mistral; "" = off
  extraArgs: [] # extra `vllm serve` flags, e.g. ["--max-num-seqs", "32"]

Credentials

Both are names of opaque secrets you create first (see Prerequisites); the key values never enter the Helm release.

apiKey:
  # REQUIRED opaque secret (encoding: plain) holding the API key clients send as a Bearer token.
  # It must EXIST BEFORE INSTALL — the deployment waits silently (no logs) otherwise. Create it with:
  # printf '%s' "$(openssl rand -hex 32)" | cpln secret create-opaque --name my-vllm-api-key --encoding plain -f -
  secretName: my-vllm-api-key
huggingFace:
  tokenSecretName: "" # opaque secret with a Hugging Face token, needed for gated models (e.g. my-vllm-hf-token); empty = anonymous

GPU

builtin uses Control Plane’s GPU locations; byok runs on your own NVIDIA nodes on a BYOK location (see GPU Cluster Setup (BYOK) for the cluster requirements). The built-in form cannot target BYOK nodes.

gpu:
  type: builtin # options: builtin (Control Plane GPU locations), byok (your own GPU cluster on a BYOK location)
  model: a10g # builtin only. options: a10g (24 GB), t4 (16 GB)
  quantity: 1 # GPUs per replica. a10g: 1, t4: 1-4, byok: 1-8; more than 1 enables tensor parallelism

Resources and Scaling

The chart refuses CPU and memory combinations the GPU type does not allow; the comments list the limits. Each replica is an independent server with its own GPUs and its own model cache.

# builtin a10g: cpu 6000m-8000m, memory 8Gi-62Gi. t4: cpu 2000m-8000m and memory 7Gi-31Gi per GPU (4000m per GPU with 2+ GPUs or from 14Gi).
# Any type: memory below 8 MiB per millicore of cpu (e.g. under 16Gi at 2000m).
# byok: fit your GPU node, e.g. cpu 2000m / memory 8Gi on a 4-vCPU g5.xlarge or g6.xlarge.
resources:
  cpu: 6000m
  memory: 16Gi

replicas: 1 # each replica runs its own GPU(s) and model cache; 2+ keeps serving through restarts and upgrades
startupTimeoutSeconds: 900 # seconds from container start to ready (download + load) before the platform restarts it; 300-1830

Model Cache

capacity sizes new volumes only; to grow a live one, see Growing the Model Cache.

volumeset:
  capacity: 50 # GiB per replica for new volumes; at least 2x the model's weights plus 5 (minimum 10)

Access

The API key guards /v1 whether or not public access is on.

publicAccess:
  enabled: false # true = the API is reachable from the internet (the API key still applies to /v1)

internalAccess:
  type: same-gvc # options: none, same-gvc, same-org, workload-list
  workloads: [] # used with workload-list
  # workloads:
  #   - //gvc/GVC_NAME/workload/WORKLOAD_NAME

Connecting

What	Address	Notes
Internal API	`http://RELEASE_NAME-vllm.GVC_NAME.cpln.local:8000/v1`	From workloads that `internalAccess.type` allows
Public API	`status.canonicalEndpoint` + `/v1` (it already includes `https://`)	Only with `publicAccess.enabled: true`; read the endpoint with `cpln workload get RELEASE_NAME-vllm --gvc GVC_NAME -o yaml`
From your machine	`cpln port-forward RELEASE_NAME-vllm 8000:8000 --gvc GVC_NAME`, then `http://localhost:8000/v1`	Works with public access off
API key	The payload of your `apiKey.secretName` secret	Read it with `cpln secret reveal API_KEY_SECRET_NAME -o yaml`
Model name	`model.servedModelName`, or the model id when it is empty	The value clients send as `"model"`

With the port-forward running and the key in VLLM_API_KEY, send a chat request with the default model:

curl http://localhost:8000/v1/chat/completions \
  -H "Authorization: Bearer $VLLM_API_KEY" -H 'Content-Type: application/json' \
  -d '{"model":"Qwen/Qwen2.5-1.5B-Instruct","messages":[{"role":"user","content":"Hello"}]}'

Any OpenAI SDK works with base_url set to the /v1 address and api_key set to the key. The LiteLLM template can also use vLLM as an OpenAI-compatible backend.

Pairing with Open WebUI

For a chat UI, install the Open WebUI template (1.3.0 or later) in the same GVC and point its OpenAI-compatible backend at http://RELEASE_NAME-vllm.GVC_NAME.cpln.local:8000/v1, reusing your vLLM API key secret. The exact values are on the Open WebUI page under Pairing with vLLM. On the vLLM side:

Keep model.toolCallParser set and matched to your model. Open WebUI offers tools on every chat, so without a parser each message fails with "auto" tool choice requires --enable-auto-tool-choice.
The default model and its 8192-token context are fine for chat.
Open WebUI’s Control Plane tools need a context of at least 64k tokens. Qwen2.5-7B-Instruct reaches 64k with YaRN:

model:
  name: Qwen/Qwen2.5-7B-Instruct
  maxModelLen: 65536
  extraArgs:
    - --hf-overrides
    - '{"max_position_embeddings":65536,"rope_parameters":{"rope_type":"yarn","factor":2.0,"original_max_position_embeddings":32768,"rope_theta":1000000.0}}'

vLLM must also allow Open WebUI as a caller; the default internalAccess.type: same-gvc does.

Operations

Backing Up

There is nothing to back up. The volume holds only downloaded model weights and vLLM’s compile cache, which a replica downloads or rebuilds on start, so the chart keeps no snapshots.

Rotating the API Key

A running replica keeps the old key until it restarts. Replace the secret’s value, then force a redeployment. Save this as rotate.yaml:

kind: secret
name: API_KEY_SECRET_NAME
type: opaque
data:
  encoding: plain
  payload: NEW-VALUE

cpln apply -f rotate.yaml
cpln secret reveal API_KEY_SECRET_NAME -o yaml
cpln workload force-redeployment RELEASE_NAME-vllm --gvc GVC_NAME

With one replica the API is unavailable while the replica restarts; with two or more it keeps serving.

Growing the Model Cache

Changing volumeset.capacity does not resize an existing volume; it only sizes new ones. Grow a live volume online, with no restart, once per replica (--volume-index 0 for the first replica, 1 for the second, and so on):

cpln volumeset expand RELEASE_NAME-vllm-models --gvc GVC_NAME --location LOCATION --volume-index 0 --new-size 100

Freeing Old Model Weights

Switching model.name leaves the old model’s weights on the volume. Remove them with the Hugging Face CLI in the image, replacing ORG/NAME with the old model id:

cpln workload exec RELEASE_NAME-vllm --gvc GVC_NAME --container vllm -- hf cache rm model/ORG/NAME -y

Deleting the model’s folder by hand frees nothing, because the weights live in a shared blob store.

Scaling and Availability

With replicas: 1 (the default) the API is down whenever the replica restarts, which includes every upgrade and setting change. Set replicas: 2 for zero-downtime restarts and upgrades; replicas are independent servers behind the workload’s load-balanced endpoint and are replaced one at a time. Each replica needs its own GPU and its own model cache volume, so check your GPU quota first. gpu.quantity above 1 splits the model across several GPUs in one replica (tensor parallelism) rather than adding replicas; on built-in GPUs only the T4 allows it.

Troubleshooting

The deployment waits and cpln logs shows nothing

Symptom: The workload never becomes ready and cpln logs returns no lines.Cause: The secret named in apiKey.secretName (or huggingFace.tokenSecretName) does not exist, so the container never starts.Fix: Read status.versions[].message, which names the missing secret, then create it as in Prerequisites. The deployment recovers by itself once it exists:

cpln workload get-deployments RELEASE_NAME-vllm --gvc GVC_NAME -o yaml

A gated model restarts with exit code 1

Symptom: The replica restarts with exit code 1, and the logs show GatedRepoError.Cause: A 401 means no token was supplied (huggingFace.tokenSecretName is empty). A 403 with “not in the authorized list” means the account that owns the token has not been granted access to the model.Fix: Accept the model’s license on huggingface.co with the token’s account, wait for access to be granted, and set huggingFace.tokenSecretName. To find the error:

cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-vllm"} |= "GatedRepoError"' --limit 50 --since 10m

Chat requests fail with an auto tool choice error

Symptom: Every message from Open WebUI or an agent client fails with "auto" tool choice requires --enable-auto-tool-choice.Cause: model.toolCallParser is empty, or does not match the model family.Fix: Set model.toolCallParser to the parser for your model: hermes (Qwen), llama3_json (Llama 3) or mistral.

The replica restarts before it becomes ready

Symptom: A large model never finishes loading; the replica restarts and starts the download or load again.Cause: The replica was not ready within startupTimeoutSeconds.Fix: Raise startupTimeoutSeconds (up to 1830) for large models.

A replica stays pending

Symptom: The replica is never scheduled.Cause: The requested CPU and memory do not fit the GPU node. On a BYOK 4-vCPU node the built-in default of 6 CPU cannot fit. On a built-in T4, larger requests the chart accepts (such as the 6000m / 16Gi default) did not schedule in testing.Fix: On BYOK, size resources to the node (about cpu: 2000m, memory: 8Gi on a g5.xlarge or g6.xlarge). On a T4, use about cpu: 3000m and memory: 12Gi.

The install fails with a GPU runtime class error on BYOK

Symptom: The deployment reports The GPU runtime class nvidia is not supported in this cluster.Cause: The cluster has no RuntimeClass named nvidia, or BYOK’s actuator has not seen it yet.Fix: Create the RuntimeClass with your cluster’s handler (see GPU Cluster Setup (BYOK)), then restart the actuator:

kubectl rollout restart deploy/actuator -n controlplane

The model volume never attaches on a BYOK GPU node

Symptom: The replica stays in Init and Kubernetes events show FailedAttachVolume.Cause: The GPU node cannot attach BYOK’s default disk type: Hyperdisk on a GKE N1 machine, or Premium SSD v2 on a non-zonal AKS node.Fix: On AKS, put the GPU node pool in an availability zone. On GKE N1, replace BYOK’s storage classes as described in GPU Cluster Setup (BYOK).

Port-forward returns 502 on a BYOK location

Symptom: cpln port-forward returns 502 Unable to connect to upstream workload.Cause: On a BYOK location, port-forward needs internalAccess.type other than none.Fix: Set internalAccess.type to same-gvc (the default) or another allowed type.

Important Notes

Create the API key secret before installing. A missing secret stalls the install with no logs; status.versions[].message names it.
The API key protects only /v1. vLLM leaves /health, /metrics, /tokenize and /detokenize unauthenticated, so with public access on anyone can read metrics and use the tokenizer.
Use a GVC with exactly one location. Every location runs its own copy with its own GPUs, and on BYOK the only location must be the BYOK location.
Rotating the API key needs a forced redeployment. The running server keeps the old key until it restarts — see Rotating the API Key.
With one replica the API is down during every restart and upgrade. Set replicas: 2 for zero downtime; each replica needs its own GPU.
volumeset.capacity only sizes new volumes, and switching models keeps the old weights. See Growing the Model Cache and Freeing Old Model Weights.

External References

vLLM on GitHub

Source, releases, and issues for the vLLM project

OpenAI-Compatible Server

The API this template serves, including tool calling

Engine Arguments

Every vllm serve flag you can pass through model.extraArgs

Supported Models

Model architectures vLLM can serve

vLLM Template

View the source files, default values, and chart definition

Template Catalog

Templates

Overview

What Gets Created

Prerequisites

GPU Cluster Setup (BYOK)

Installation

UI

CLI

Terraform

Pulumi

Configuration

Image

Model

Credentials

GPU

Resources and Scaling

Model Cache

Access

Connecting

Pairing with Open WebUI

Operations

Backing Up

Rotating the API Key

Growing the Model Cache

Freeing Old Model Weights

Scaling and Availability

Troubleshooting

Important Notes

External References

vLLM on GitHub

OpenAI-Compatible Server

Engine Arguments

Supported Models

vLLM Template

​Overview

​What Gets Created

​Prerequisites

​GPU Cluster Setup (BYOK)

​Installation

UI

CLI

Terraform

Pulumi

​Configuration

​Image

​Model

​Credentials

​GPU

​Resources and Scaling

​Model Cache

​Access

​Connecting

​Pairing with Open WebUI

​Operations

​Backing Up

​Rotating the API Key

​Growing the Model Cache

​Freeing Old Model Weights

​Scaling and Availability

​Troubleshooting

​Important Notes

​External References

vLLM on GitHub

OpenAI-Compatible Server

Engine Arguments

Supported Models

vLLM Template

Overview

What Gets Created

Prerequisites

GPU Cluster Setup (BYOK)

Installation

Configuration

Image

Model

Credentials

GPU

Resources and Scaling

Model Cache

Access

Connecting

Pairing with Open WebUI

Operations

Backing Up

Rotating the API Key

Growing the Model Cache

Freeing Old Model Weights

Scaling and Availability

Troubleshooting

Important Notes

External References