CPLN Task Runner

Overview

Control Plane Task Runner is a self-hosted HTTP task queue and scheduler, similar to Google Cloud Tasks. You enqueue a task over HTTP and the workers deliver it to its target URL, with automatic retries, delayed and scheduled execution, per-client rate limiting and a circuit breaker. This template deploys the API, the workers, and the Redis Sentinel cluster they persist to.

This template deploys into an existing GVC that you already have. It does not create, provision or manage a GVC.

/v1/enqueue has no authentication, which is why api.public.enabled defaults to false. An unknown client_id is accepted and auto-registered rather than rejected, so a client ID is not a credential. With public access on, anyone who finds the endpoint can make a worker send outbound HTTP requests with a method, headers and body of their choosing. See Public Access and the Open Queue.

What Gets Created

Resource	Name	Purpose	Optional
Workload	`RELEASE_NAME-task-runner-api`	HTTP API on port 8080 for enqueuing tasks and the `/admin/*` endpoints (serverless, autoscales 1 to 3)	`api.enabled: true` (default)
Workload	`RELEASE_NAME-task-runner-worker`	Delivers queued tasks to their target URLs; health port 8082 (serverless, autoscales 1 to 5)	`worker.enabled: true` (default)
Workload	`RELEASE_NAME-redis`	Redis task store (stateful, 3 replicas)	—
Workload	`RELEASE_NAME-sentinel`	Redis Sentinel on port 26379, tracks the Redis primary (stateful, 3 replicas)	—
Volume set	`RELEASE_NAME-redis-vs`	Redis data, daily snapshots kept 7 days	`redis.redis.persistence.enabled: true` (default)
Volume set	`RELEASE_NAME-sentinel-vs`	Sentinel state	`redis.sentinel.persistence.enabled: true` (default)
Secret	`RELEASE_NAME-redis-config`, `RELEASE_NAME-sentinel-config`	Redis and Sentinel configuration	—
Identity	`RELEASE_NAME-task-runner-identity`	Identity used by the API and worker workloads	—
Policy	`RELEASE_NAME-task-runner-policy`	Grants that identity `reveal` on your Redis and Sentinel credentials secrets and on your admin key secret, nothing else	—
Identity and policy	`RELEASE_NAME-redis-identity`, `RELEASE_NAME-sentinel-identity` and matching `-policy`	Grant Redis and Sentinel `reveal` on their own configuration and password secrets	—

The policies grant only the reveal permission on secrets. The template’s identities have no cloud access and no permission on any other Control Plane resource.

Prerequisites

Three secrets must exist before you install: the Redis password, the Sentinel password and the admin API key. Secrets are org-level, so no GVC flag is involved.

Create the Redis and Sentinel credentials secrets

Two dictionary secrets, each with the key password: one holds the bundled Redis password, the other the Sentinel password. Redis, Sentinel, the API and the worker read both. The names below are the chart defaults for redis.redis.auth.fromSecret.name and redis.sentinel.auth.fromSecret.name:

cpln secret create-dictionary --name my-cpln-task-runner-redis-credentials --entry password="$(openssl rand -hex 24)"
cpln secret create-dictionary --name my-cpln-task-runner-sentinel-credentials --entry password="$(openssl rand -hex 24)"

Upgrading from 1.3.0? Do not generate new passwords; see Upgrading From 1.3.0.

Create the admin key secret

The /admin/* endpoints create, edit and delete clients and rate-limit tiers, and they are guarded by the X-Admin-Key header. The key is an opaque secret with encoding plain whose payload is the key itself:

printf '%s' "$(openssl rand -hex 32)" | cpln secret create-opaque --name SECRET_NAME --encoding plain -f -

Use printf, not echo. A trailing newline from echo becomes part of the key and then has to be sent in every admin request. Set api.admin.apiKeySecretName to this name.

Read the key back later

Pass -o yaml. A bare cpln secret reveal prints only a summary table, not the payload:

cpln secret reveal SECRET_NAME -o yaml

Create the secrets before installing, or the deployment wedges silently. An install that names a secret which does not exist still reports success, and every workload that reads it never starts. The container never runs, so cpln logs returns zero lines. The one place the reason appears is status.versions[].message:

cpln workload get-deployments RELEASE_NAME-task-runner-api --gvc GVC_NAME -o yaml

The secret SECRET_NAME no longer exists. Workload updates are paused until the
secret is added or the reference to the secret removed.

Use get-deployments. Plain cpln workload get has no versions key. Run the same command for RELEASE_NAME-redis, RELEASE_NAME-sentinel and RELEASE_NAME-task-runner-worker when a credentials secret is missing. Once the secret exists the workload recovers on its own after several minutes, or run cpln workload force-redeployment RELEASE_NAME-task-runner-api --gvc GVC_NAME to skip the wait.

Installation

Create the three secrets first, then install with the admin key secret’s name. If you named the credentials secrets differently from the defaults, also set redis.redis.auth.fromSecret.name and redis.sentinel.auth.fromSecret.name:

cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/cpln-task-runner \
  --version 1.4.0 --gvc GVC_NAME \
  --set api.admin.apiKeySecretName=SECRET_NAME

For the other install methods, follow the instructions for your preferred tool:

UI

Browse, install, and manage templates visually

CLI

Manage templates from your terminal

Terraform

Declare templates in your Terraform configurations

Pulumi

Declare templates in your Pulumi programs

Configuration

Image

The same image runs both the API and the worker:

image: controlplanecorporation/cpln-task-runner:0.4

API Workload

api:
  enabled: true              # false = no API workload
  replicas:
    min: 1                   # autoscaling range
    max: 3
  port: 8080
  public:
    enabled: false           # DEFAULT. /v1/enqueue has no authentication; read Public Access and the Open Queue first
    pathPrefix: ""           # not used for routing by this chart version
  admin:
    apiKeySecretName: my-cpln-task-runner-admin-key   # REQUIRED: opaque secret whose payload is the X-Admin-Key value; "" is rejected while public.enabled is true
  resources:
    cpu: 500m
    memory: 512Mi
  env:
    logLevel: info           # debug, info, warn, error
    otelEndpoint: ""         # OpenTelemetry endpoint; empty disables tracing
    connectRetries: 30       # Redis connection attempts at startup
    retryIntervalSec: 2      # seconds between those attempts

Setting api.admin.apiKeySecretName to "" turns admin authentication off entirely. The chart permits that only while api.public.enabled is false.

Worker Workload

worker:
  enabled: true              # false = no worker workload
  replicas:
    min: 1                   # autoscaling range
    max: 5
  port: 8082                 # health checks only
  resources:
    cpu: 1
    memory: 1Gi
  env:
    logLevel: info
    concurrency: 10          # concurrent tasks per replica
    taskTimeoutSec: 1800     # per-task timeout in seconds
    maxRetry: 5              # maximum delivery attempts
    allowPrivateUrls: false  # true lets tasks target private and internal URLs
    cbFailureThreshold: 5    # circuit breaker: failures before it opens
    cbTimeoutSec: 30         # circuit breaker: seconds before it retries
    connectRetries: 30
    retryIntervalSec: 2
    otelEndpoint: ""

Workers fetch whatever URL a task carries. allowPrivateUrls: false is what keeps an enqueued task off internal addresses.

Secrets and Redis

Redis and Sentinel read their passwords from the prerequisite credentials secrets:

redis:
  redis:
    auth:
      fromSecret:
        enabled: true          # must stay true
        name: my-cpln-task-runner-redis-credentials   # prerequisite dictionary secret for Redis
        passwordKey: password  # key holding the Redis password
    persistence:
      enabled: true            # false = no volume set, Redis data is not persisted
  sentinel:
    auth:
      fromSecret:
        enabled: true          # must stay true
        name: my-cpln-task-runner-sentinel-credentials   # prerequisite dictionary secret for Sentinel
        passwordKey: password  # key holding the Sentinel password
    persistence:
      enabled: true

The admin API key is never in these secrets. It is always its own opaque secret, named by api.admin.apiKeySecretName.

Connecting

What	Address	Notes
API (internal, default)	`RELEASE_NAME-task-runner-api.GVC_NAME.cpln.local:8080`	Reachable from workloads in the same GVC. No credential for `/v1/`; `X-Admin-Key` for `/admin/`
API (public, opt-in)	the workload’s canonical endpoint	Only with `api.public.enabled: true`. Read it from `status.canonicalEndpoint`
Redis Sentinel	`RELEASE_NAME-sentinel.GVC_NAME.cpln.local:26379`	Master name `mymaster`; used by the API and worker. Password is `password` in your Sentinel credentials secret
Admin key	your opaque secret	`cpln secret reveal SECRET_NAME -o yaml`
Redis and Sentinel passwords	your two credentials secrets	`cpln secret reveal my-cpln-task-runner-redis-credentials -o yaml` (and the same for the Sentinel secret)

Read the public endpoint, when enabled:

cpln workload get RELEASE_NAME-task-runner-api --gvc GVC_NAME -o yaml

To reach the private API from your machine, open a tunnel and check readiness. This uses the same /health/ready path as the workload readiness probe:

cpln port-forward RELEASE_NAME-task-runner-api 8080:8080 --gvc GVC_NAME
curl http://127.0.0.1:8080/health/ready

To see what happened to a delivered task, read the worker logs. The second form filters on the server, so put any text you expect in the line between the quotes:

cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-task-runner-worker"}' --limit 50 --since 10m
cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-task-runner-worker"} |= "ERROR"' --limit 50 --since 10m

Enqueuing Tasks

Post a task to /v1/enqueue. With the default private API, send it from a workload in the same GVC to http://RELEASE_NAME-task-runner-api.GVC_NAME.cpln.local:8080, or through the tunnel above to http://127.0.0.1:8080:

curl -X POST http://127.0.0.1:8080/v1/enqueue \
  -H "Content-Type: application/json" \
  -d '{
    "client_id": "my-service",
    "queue": "default",
    "task": {
      "url": "https://api.example.com/webhook",
      "method": "POST",
      "headers": {"Content-Type": "application/json"},
      "body": "{\"event\": \"user.created\"}"
    }
  }'

This request carries no credential, and none exists.

Admin Endpoints

Every /admin/* request needs the X-Admin-Key header, whose value is the payload of your api.admin.apiKeySecretName secret:

# List clients
curl http://127.0.0.1:8080/admin/clients \
  -H "X-Admin-Key: YOUR-ADMIN-KEY"

# Create or update a client
curl -X POST http://127.0.0.1:8080/admin/clients/set \
  -H "X-Admin-Key: YOUR-ADMIN-KEY" \
  -H "Content-Type: application/json" \
  -d '{"client_id": "new-service", "tier": "premium", "enabled": true}'

Rate Limiting

Rate limits apply per client, and a client tier is assigned through the admin API:

Tier	Requests/min	Max Concurrent
`free`	10	1
`basic`	100	5
`premium`	1,000	20
`enterprise`	5,000	50

OpenTelemetry

Set otelEndpoint under api.env and worker.env to export traces. In the GVC settings, set the Tracing Provider to Control Plane, then use the built-in HTTP collector endpoint:

tracing.controlplane:4318

Operations

Backing Up

The Redis volume set RELEASE_NAME-redis-vs takes a snapshot every day and keeps each one for 7 days. List the snapshots:

cpln volumeset snapshot get RELEASE_NAME-redis-vs --gvc GVC_NAME

Restoring a Backup

A restore procedure for this template has not been verified. Do not rely on the snapshots as a tested recovery path for queued tasks.

Rotating the Admin Key

Apply the whole secret with a new payload, confirm it, then force a redeployment of the API. A running replica keeps the old key until it is redeployed:

cat > rotate.yaml <<'EOF'
kind: secret
name: SECRET_NAME
type: opaque
data:
  encoding: plain
  payload: NEW-VALUE
EOF
cpln apply -f rotate.yaml
cpln secret reveal SECRET_NAME -o yaml
cpln workload force-redeployment RELEASE_NAME-task-runner-api --gvc GVC_NAME

Every admin caller must then send the new key.

Rotating the Redis and Sentinel Passwords

Redis reads its password from the secret at every start, but Sentinel keeps the passwords it first started with on its volume, so editing the secrets alone locks the API and worker out. Apply the new passwords live on every node first, then restart the API and worker. rx and sx run a command on one Redis node or Sentinel (LOCATION is your GVC’s location; nodes and Sentinels are numbered from 0 up to redis.redis.replicas and redis.sentinel.replicas minus one).

Accept the new passwords alongside the old ones on every Redis node and Sentinel:

rx() { cpln workload exec RELEASE_NAME-redis --gvc GVC_NAME --location LOCATION \
  --container redis --replica "RELEASE_NAME-redis-$1" -- "${@:2}"; }
sx() { cpln workload exec RELEASE_NAME-sentinel --gvc GVC_NAME --location LOCATION \
  --container sentinel --replica "RELEASE_NAME-sentinel-$1" -- "${@:2}"; }
# The loops cover the default 3 Redis nodes and 3 Sentinels; list one number per node if you changed the counts
# On every Redis node: accept the new Redis password alongside the old one, and replicate with it
for n in 0 1 2; do
  rx $n redis-cli --no-auth-warning -a OLD_REDIS_PASSWORD ACL SETUSER default ">NEW_REDIS_PASSWORD"
  rx $n redis-cli --no-auth-warning -a OLD_REDIS_PASSWORD CONFIG SET masterauth NEW_REDIS_PASSWORD
done
# On every Sentinel: monitor Redis with the new Redis password, and accept the new Sentinel password alongside the old one
for n in 0 1 2; do
  sx $n redis-cli -p 26379 --no-auth-warning -a OLD_SENTINEL_PASSWORD SENTINEL SET mymaster auth-pass NEW_REDIS_PASSWORD
  sx $n redis-cli -p 26379 --no-auth-warning -a OLD_SENTINEL_PASSWORD ACL SETUSER default ">NEW_SENTINEL_PASSWORD"
done

Put the new passwords in the two secrets (key password). For each secret, export it, change only password, apply it and read it back:

cpln secret reveal my-cpln-task-runner-redis-credentials -o yaml > secret.yaml
# edit only the password key in secret.yaml
cpln apply -f secret.yaml
cpln secret reveal my-cpln-task-runner-redis-credentials -o yaml

Restart the API and worker, which keep the old passwords until they restart:

cpln workload force-redeployment RELEASE_NAME-task-runner-api --gvc GVC_NAME
cpln workload force-redeployment RELEASE_NAME-task-runner-worker --gvc GVC_NAME

Once get-deployments shows both at a higher version, alone and ready, retire the old passwords and check that the Sentinels can still fail over:

for n in 0 1 2; do rx $n redis-cli --no-auth-warning -a NEW_REDIS_PASSWORD ACL SETUSER default "<OLD_REDIS_PASSWORD"; done
# Sentinels authenticate to each other with this password; without it they lose quorum once the old one is retired
for n in 0 1 2; do sx $n redis-cli -p 26379 --no-auth-warning -a NEW_SENTINEL_PASSWORD SENTINEL CONFIG SET sentinel-pass NEW_SENTINEL_PASSWORD; done
# FLUSHCONFIG writes Sentinel's own password change to its volume, so it survives a restart
for n in 0 1 2; do
  sx $n redis-cli -p 26379 --no-auth-warning -a NEW_SENTINEL_PASSWORD ACL SETUSER default "<OLD_SENTINEL_PASSWORD"
  sx $n redis-cli -p 26379 --no-auth-warning -a NEW_SENTINEL_PASSWORD SENTINEL FLUSHCONFIG
done
# Prints OK 3 usable Sentinels
sx 0 redis-cli -p 26379 --no-auth-warning -a NEW_SENTINEL_PASSWORD SENTINEL CKQUORUM mymaster

If both fromSecret.name values point at one secret, edit that one secret in step 2, each password under its own key.

Upgrading From 1.2.x

Two defaults change, and both break an existing workflow that relied on the old behaviour.

	`1.2.x`	`1.3.0`
Admin API key	`api.env.adminApiKey`, a value shipped empty, which disabled admin authentication	`api.admin.apiKeySecretName`, a required opaque secret you create
Public access	`api.public.enabled: true`	`api.public.enabled: false`
Bundled Redis passwords	A working published value	`change-me-` placeholders
`redis.admin.fromSecret`	Present	Removed. The admin key is its own secret whatever `createSecret` is set to

An upgrade that still carries api.env.adminApiKey or redis.admin.fromSecret, or an empty apiKeySecretName while public access is on, is rejected at render before anything is applied.

Create the admin key secret

Follow Prerequisites. Put the key you use today into it so existing admin callers keep working.

Drop the removed keys from your values

Remove api.env.adminApiKey and redis.admin.fromSecret, and set api.admin.apiKeySecretName instead.

Decide about public access

api.public.enabled now defaults to false. Read Public Access and the Open Queue before setting it back to true.

Upgrade

cpln helm upgrade RELEASE_NAME oci://ghcr.io/controlplane-com/templates/cpln-task-runner \
  --version 1.3.0 --gvc GVC_NAME \
  --set api.admin.apiKeySecretName=SECRET_NAME

The first upgrade after an install re-applies the bundled Redis even with identical values, which restarts it, and the API returns errors briefly while Redis comes back. Later upgrades are clean.

Then follow Upgrading From 1.3.0: the Redis and Sentinel passwords your 1.2.x release runs with (mypassword if never changed) go into the two credentials secrets.

Upgrading From 1.3.0

1.4.0 no longer takes the Redis and Sentinel passwords as values, and no longer creates the task-runner-secrets secret. Use the CURRENT passwords: Redis and Sentinel keep the ones they started with, so new ones would lock the API and worker out. To change them, upgrade first, then follow Rotating the Redis and Sentinel Passwords. With createSecret: true (the default), the upgrade deletes task-runner-secrets (or the secret named by your secretName, if you changed it), so copy the passwords into two new secrets before upgrading:

Read the current passwords

redis-password is the Redis password, redis-sentinel-password the Sentinel password. Use the name you set in secretName if you changed it:

cpln secret reveal task-runner-secrets -o yaml

Create the two credentials secrets with those values

cpln secret create-dictionary --name my-cpln-task-runner-redis-credentials --entry password='YOUR-CURRENT-REDIS-PASSWORD'
cpln secret create-dictionary --name my-cpln-task-runner-sentinel-credentials --entry password='YOUR-CURRENT-SENTINEL-PASSWORD'

Drop the removed keys from your values

Remove createSecret, secretName, redis.redisPassword and redis.sentinelPassword, and point redis.redis.auth.fromSecret.name and redis.sentinel.auth.fromSecret.name at the new secrets (the defaults above). The chart refuses to render while any of the removed keys is still set, and refuses the name task-runner-secrets; do not reuse a custom secretName either, because the upgrade deletes that secret too.

Upgrade

cpln helm upgrade RELEASE_NAME oci://ghcr.io/controlplane-com/templates/cpln-task-runner \
  --version 1.4.0 --gvc GVC_NAME \
  --set api.admin.apiKeySecretName=SECRET_NAME

With createSecret: false and a secret of your own, keep it: remove the same four keys, keep both fromSecret.name values pointing at your secret, and set both passwordKey values to its key names explicitly (redis-password and redis-sentinel-password if you kept the 1.3.0 defaults; 1.4.0 defaults both to password). Nothing is deleted.

Public Access and the Open Queue

api.public.enabled is false by default. Turning it on exposes the following:

/v1/enqueue requires no credential. There is no header, token or signature to supply.
An unknown client_id is auto-registered, not rejected. A client ID is therefore not a secret and not a control.
The API becomes a request relay. Anyone reaching it can make a worker send outbound HTTP with the method, headers and body they choose. worker.env.allowPrivateUrls: false keeps those requests off internal addresses.
/metrics is unauthenticated and its labels list every client_id.

No setting closes this, because the application has no client authentication. If you need submissions from outside the GVC, put an authenticating proxy of your own in front of the API and leave api.public.enabled at false. Access changes can take several minutes to take effect, so re-test over that window.

Scaling and Availability

The API and worker are serverless workloads that autoscale on concurrent requests within api.replicas and worker.replicas. Raise worker.replicas.max or worker.env.concurrency for more delivery throughput. Redis runs as three replicas with three Sentinels, which elect a new primary if the current one fails.

Troubleshooting

The API workload never starts and cpln logs returns nothing

Cause: a prerequisite secret does not exist: the one named by api.admin.apiKeySecretName, or a Redis or Sentinel credentials secret. get-deployments shows

The secret SECRET_NAME no longer exists. Workload updates are paused until the secret is added or the reference to the secret removed.

Fix: create the secret as in Prerequisites. The workload recovers on its own, or run cpln workload force-redeployment RELEASE_NAME-task-runner-api --gvc GVC_NAME.

Install fails with apiKeySecretName is empty while api.public.enabled is true

Cause: api.admin.apiKeySecretName is "" and public access is on. The chart refuses to publish unauthenticated admin endpoints.Fix: set api.admin.apiKeySecretName to your admin key secret, or set api.public.enabled: false.

Install or upgrade fails with api.env.adminApiKey was REMOVED

Cause: your values still carry the 1.2.x admin key (api.env.adminApiKey or redis.admin.fromSecret).Fix: follow Upgrading From 1.2.x.

Install or upgrade fails with Removed in cpln-task-runner 1.4.0

Cause: your values still carry createSecret, secretName, redis.redisPassword or redis.sentinelPassword.Fix: follow Upgrading From 1.3.0.

Admin requests are rejected

Cause: the X-Admin-Key header is missing or does not match the secret payload, for example because the key was created with echo and carries a trailing newline, or the secret was rotated without a redeployment.Fix: read the key with cpln secret reveal SECRET_NAME -o yaml, and redeploy the API after any rotation.

Tasks to internal addresses are never delivered

Cause: worker.env.allowPrivateUrls is false, so workers refuse private and internal targets.Fix: set it to true only if you accept that any enqueued task can then reach anything the worker can route to.

Important Notes

Create the admin key secret and the Redis and Sentinel credentials secrets before installing. A missing secret wedges the workloads that read it with no log output; the reason is in status.versions[].message from cpln workload get-deployments.
Leave api.public.enabled at false unless an authenticating proxy fronts the API. /v1/enqueue is unauthenticated and auto-registers unknown client IDs.
Never change the Redis or Sentinel password by editing the secrets alone. Sentinel keeps the passwords it first started with on its volume, so that locks the API and worker out. Follow Rotating the Redis and Sentinel Passwords.
Force a redeployment after rotating the admin key. A running replica keeps the old key.

External References

CPLN Task Runner Template

View the source files, default values, and chart definition

Task Runner Image

The container image used by the API and worker

Control Plane Secrets

Secret types and how workloads reference them

OpenTelemetry Documentation

OpenTelemetry distributed tracing documentation

Template Catalog

Templates

Overview

What Gets Created

Prerequisites

Installation

UI

CLI

Terraform

Pulumi

Configuration

Image

API Workload

Worker Workload

Secrets and Redis

Connecting

Enqueuing Tasks

Admin Endpoints

Rate Limiting

OpenTelemetry

Operations

Backing Up

Restoring a Backup

Rotating the Admin Key

Rotating the Redis and Sentinel Passwords

Upgrading From 1.2.x

Upgrading From 1.3.0

Public Access and the Open Queue

Scaling and Availability

Troubleshooting

Important Notes

External References

CPLN Task Runner Template

Task Runner Image

Control Plane Secrets

OpenTelemetry Documentation

​Overview

​What Gets Created

​Prerequisites

​Installation

UI

CLI

Terraform

Pulumi

​Configuration

​Image

​API Workload

​Worker Workload

​Secrets and Redis

​Connecting

​Enqueuing Tasks

​Admin Endpoints

​Rate Limiting

​OpenTelemetry

​Operations

​Backing Up

​Restoring a Backup

​Rotating the Admin Key

​Rotating the Redis and Sentinel Passwords

​Upgrading From 1.2.x

​Upgrading From 1.3.0

​Public Access and the Open Queue

​Scaling and Availability

​Troubleshooting

​Important Notes

​External References

CPLN Task Runner Template

Task Runner Image

Control Plane Secrets

OpenTelemetry Documentation

Overview

What Gets Created

Prerequisites

Installation

Configuration

Image

API Workload

Worker Workload

Secrets and Redis

Connecting

Enqueuing Tasks

Admin Endpoints

Rate Limiting

OpenTelemetry

Operations

Backing Up

Restoring a Backup

Rotating the Admin Key

Rotating the Redis and Sentinel Passwords

Upgrading From 1.2.x

Upgrading From 1.3.0

Public Access and the Open Queue

Scaling and Availability

Troubleshooting

Important Notes

External References