Temporal

​

Overview

​

What Gets Created

​

Prerequisites

cpln secret create-dictionary --name DB_CREDENTIALS_SECRET_NAME \
  --entry username=temporal \
  --entry password="$(openssl rand -hex 24)" \
  --entry database=temporal

• database must be temporal. The server reads its main store from the database named temporal, while PostgreSQL creates the one the secret names. With any other value both Temporal containers stop at boot and log CPLN_TEMPORAL_DATABASE_GUARD.
• Set its name in the key of the database mode you use: postgresHA.config.credentialsSecretName (HA, the default) or postgres.config.credentialsSecretName (single-instance). Secret names are org-wide, so give each Temporal release its own secret.
• PostgreSQL reads the secret only when its data is first initialized — see Rotating the Database Password.

cpln workload get-deployments RELEASE_NAME-temporal --gvc GVC_NAME -o yaml

​

Installation

cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/temporal \
  --version 2.1.0 --gvc GVC_NAME \
  --set postgresHA.config.credentialsSecretName=DB_CREDENTIALS_SECRET_NAME

​

Configuration

​

Temporal Server

image: temporalio/server:1.32.1 # server binary; configured from env (archival adds a generated config file)

replicas: 1 # 1-5; 3 survives a replica loss. >1 rolling restarts are noisy — see README

resources:
  maxCpu: 1000m
  maxMemory: 2Gi
  minCpu: 500m
  minMemory: 1Gi

adminTools:
  image: temporalio/admin-tools:1.32.1 # schema migrations + `temporal` CLI sidecar; same tag as image

historyShards: 512 # fixed at first install; a later change is ignored (the cluster keeps its original count)

namespaceRetention: 72h # retention of the `default` namespace, applied when it is first created

• image / adminTools.image — The Temporal server and the admin sidecar that runs alongside it in every replica. Both must carry the same tag, and the chart accepts only tags in the 1.30, 1.31 and 1.32 minors. Temporal upgrades one minor version at a time; see Upgrading from 1.x.
• replicas — Every replica runs the full server, and together they form one Temporal cluster. See Scaling and Availability before changing it.
• historyShards — Fixed when the cluster is first created. A later change has no effect: the server keeps the original count. 512 suits most deployments, so size it before installing.
• namespaceRetention — How long closed workflow histories stay queryable in the default namespace, as a Go duration such as 72h or 168h. It is applied only when the namespace is first created; a later change does not alter an existing namespace.

​

Web UI

ui:
  enabled: true
  image: temporalio/ui:2.55.0
  allowedCIDRs: [] # IP ranges that may open the UI in a browser (e.g. 203.0.113.10/32); empty = internal-only. The UI has no login
  resources:
    maxCpu: 500m
    maxMemory: 512Mi
    minCpu: 125m
    minMemory: 128Mi

• ui.enabled — Set false to remove the UI workload.
• ui.allowedCIDRs — IP ranges that may open the UI in a browser on its canonical endpoint, such as your office or VPN egress address (203.0.113.10/32). Empty keeps the UI internal-only. IPv4 ranges must be /8 or narrower and IPv6 ranges /32 or narrower; the chart refuses broader ranges, including the whole internet. A change takes up to a few minutes to apply. Only the UI is opened; the server stays internal-only.

​

Access

internalAccess: # internal firewall scope for the server (gRPC :7233, HTTP API :7243, cluster ports) and the UI (:8080)
  type: same-gvc # options: none, same-gvc, same-org, workload-list
  workloads: [] # used with workload-list; the server and UI workloads are always added
  # workloads:
  #   - //gvc/GVC_NAME/workload/WORKLOAD_NAME

​

Archival

archival:
  enabled: false # declares the S3 archiver on the server and binds the identity to AWS
  defaultNamespace: true # archive the `default` namespace (false = stop archiving it)
  s3:
    bucket: my-temporal-archival-bucket
    region: us-east-1
    prefix: temporal # folder in the bucket; history and visibility go below it
    cloudAccountName: my-s3-cloud-account
    policyName: my-temporal-archival-policy # AWS IAM policy scoped to the bucket

• archival.enabled — Declares the S3 archiver on the server and binds the server identity to your AWS cloud account and IAM policy. The chart refuses it on a 1.30 image; when upgrading from 1.x, turn it on after the step to 1.31.
• archival.defaultNamespace — Whether the default namespace archives. With archival on, default archives to s3://BUCKET/PREFIX/history and s3://BUCKET/PREFIX/visibility, including on an existing install.
• archival.s3.* — The bucket, its region, the folder in the bucket, the cloud account and the IAM policy from the setup steps.

cpln workload exec RELEASE_NAME-temporal --gvc GVC_NAME --container admin -- temporal workflow list -n default --archived
cpln workload exec RELEASE_NAME-temporal --gvc GVC_NAME --container admin -- temporal workflow show -n default -w WORKFLOW_ID -r RUN_ID

• Temporal never changes a namespace’s archival target once it is set, so a later change to bucket or prefix applies only to namespaces that do not archive yet.
• Other namespaces can archive to the same bucket: create them with temporal operator namespace create -n NAMESPACE --history-archival-state enabled --history-uri s3://BUCKET/PREFIX/history --visibility-archival-state enabled --visibility-uri s3://BUCKET/PREFIX/visibility.
• The server identity keeps its AWS binding until the release is uninstalled.

​

Archival Storage Setup

{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow", "Action": ["s3:ListBucket"], "Resource": ["arn:aws:s3:::YOUR_BUCKET"] },
    { "Effect": "Allow", "Action": ["s3:PutObject", "s3:GetObject"], "Resource": ["arn:aws:s3:::YOUR_BUCKET/YOUR_PREFIX/*"] }
  ]
}

​

PostgreSQL

​

Highly Available PostgreSQL

postgresHA:
  enabled: true
  image: controlplanecorporation/patroni-postgres:0.8 # pgvector 0.8.7 (CVE-2026-103484); tags 0.1-0.7 are refused
  config:
    # REQUIRED PREREQUISITE SECRET (HA mode) — CREATE IT BEFORE YOU INSTALL. If
    # it does not exist the deployment WEDGES silently and looks like a platform
    # fault. A `dictionary` secret holding exactly three keys: username, password,
    # database — and `database` MUST be `temporal` (the server's main store; any
    # other value stops the admin container at boot). PostgreSQL reads it once, when
    # its data is first initialized, and Temporal connects with the same keys. See
    # README Prerequisites for the command.
    credentialsSecretName: my-temporal-db-credentials
  replicas: 3
  volumeset:
    capacity: 10 # initial capacity in GiB per replica (minimum is 10)

  backup: # optional database backups — see README storage setup
    enabled: false
    mode: logical # logical or wal-g
    resources:
      cpu: 100m
      memory: 512Mi # 128Mi OOMKills wal-g mid-backup
    logical:
      image: ghcr.io/controlplane-com/backup-images/postgres-backup:17.1.0
      schedule: "0 2 * * *"
    walg:
      intervalSeconds: 21600
    provider: aws # options: aws, gcp, minio
    aws:
      bucket: temporal-pg-backup-bucket
      region: us-east-1
      cloudAccountName: my-s3-cloud-account
      policyName: temporal-pg-backup-policy
      prefix: postgres/backups
    gcp:
      bucket: temporal-pg-backup-bucket
      cloudAccountName: my-gcs-cloud-account
      prefix: postgres/backups
    minio:
      endpoint: http://my-minio-workload:9000
      bucket: temporal-pg-backup-bucket
      # REQUIRED PREREQUISITE SECRET when provider is `minio` — a `dictionary`
      # secret holding exactly `accessKey` and `secretKey`. See README.
      credentialsSecretName: my-temporal-minio-credentials
      prefix: postgres/backups

• postgresHA.config.credentialsSecretName — Name of the prerequisite database credentials secret the HA cluster and Temporal read (see Prerequisites). Secret names are organization-wide, so give each Temporal release its own.
• postgresHA.replicas / postgresHA.volumeset.capacity — PostgreSQL replica count and volume size per replica. The remaining cluster settings take the defaults of the Postgres Highly Available template; the HAProxy leader endpoint must stay enabled because it is the address Temporal connects to.
• postgresHA.backup.* — Optional backups of both Temporal databases, as scheduled logical dumps or continuous WAL-G archiving. Keep resources.memory at 512Mi or more for WAL-G. See Backing Up.

​

Single-Instance PostgreSQL

postgres:
  enabled: false
  config:
    # REQUIRED PREREQUISITE SECRET (single-instance mode) — the same kind of
    # `dictionary` secret as postgresHA.config.credentialsSecretName above
    # (keys: username, password, database; database MUST be `temporal`).
    # Create it BEFORE you install.
    credentialsSecretName: my-temporal-db-credentials
  volumeset:
    capacity: 10 # initial capacity in GiB (minimum is 10)

  backup: # optional database backups (Postgres 17+) — see README storage setup
    enabled: false
    image: ghcr.io/controlplane-com/backup-images/postgres-backup:18.1.0
    schedule: "0 2 * * *"
    resources:
      cpu: 100m
      memory: 128Mi
    provider: aws # options: aws, gcp, minio
    aws:
      bucket: temporal-pg-backup-bucket
      region: us-east-1
      cloudAccountName: my-s3-cloud-account
      policyName: temporal-pg-backup-policy
      prefix: postgres/backups
    gcp:
      bucket: temporal-pg-backup-bucket
      cloudAccountName: my-gcs-cloud-account
      prefix: postgres/backups
    minio:
      endpoint: http://my-minio-workload:9000
      bucket: temporal-pg-backup-bucket
      # REQUIRED prerequisite dictionary secret holding `accessKey` and `secretKey`,
      # only when this backup is enabled with provider: minio. See README.
      credentialsSecretName: my-temporal-minio-credentials
      prefix: postgres/backups

• postgres.config.credentialsSecretName — Name of the prerequisite credentials secret in single-instance mode (the HA counterpart is postgresHA.config.credentialsSecretName). Its database key must be temporal.

​

Connecting

cpln workload exec RELEASE_NAME-temporal --gvc GVC_NAME --container admin -- temporal workflow list

cpln workload get RELEASE_NAME-temporal-ui --gvc GVC_NAME -o yaml

cpln port-forward RELEASE_NAME-temporal-ui 8080:8080 --gvc GVC_NAME

​

First Run

import asyncio
from datetime import timedelta

from temporalio import activity, workflow
from temporalio.client import Client
from temporalio.worker import Worker


@activity.defn
async def say_hello(name: str) -> str:
    return f"Hello, {name}!"


@workflow.defn
class HelloWorkflow:
    @workflow.run
    async def run(self, name: str) -> str:
        return await workflow.execute_activity(
            say_hello, name, start_to_close_timeout=timedelta(seconds=10)
        )


async def main():
    client = await Client.connect(
        "RELEASE_NAME-temporal.GVC_NAME.cpln.local:7233",  # full hostname required
        namespace="default",
    )
    worker = Worker(
        client,
        task_queue="my-task-queue",
        workflows=[HelloWorkflow],
        activities=[say_hello],
    )
    await worker.run()


if __name__ == "__main__":  # required, see note below
    asyncio.run(main())

cpln workload exec RELEASE_NAME-temporal --gvc GVC_NAME --container admin \
  -- temporal workflow start --namespace default \
  --task-queue my-task-queue --type HelloWorkflow --workflow-id first-run \
  --input '"Temporal"'

​

Nexus and Worker Deployments

• Nexus works with endpoints that target a namespace and task queue in this Temporal cluster; create them with temporal operator nexus endpoint create or the UI’s Nexus page. Endpoints that target an external URL are not supported.
• Worker Deployments (worker versioning, pinned and auto-upgrade workflows) work. The UI’s Workers Instances tab relies on worker heartbeats, which need Python SDK 1.20 or later.

​

Operations

​

Backing Up

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": [
      "s3:GetObject",
      "s3:PutObject",
      "s3:DeleteObject",
      "s3:ListBucket",
      "s3:GetObjectVersion",
      "s3:DeleteObjectVersion",
      "s3:GetBucketLocation",
      "s3:AbortMultipartUpload",
      "s3:ListBucketMultipartUploads",
      "s3:ListMultipartUploadParts"
    ],
    "Resource": [
      "arn:aws:s3:::YOUR_BUCKET",
      "arn:aws:s3:::YOUR_BUCKET/*"
    ]
  }]
}

cpln secret create-dictionary --name MINIO_CREDENTIALS_SECRET_NAME \
  --entry accessKey=ACCESS_KEY \
  --entry secretKey=SECRET_KEY

cpln workload get-deployments RELEASE_NAME-postgres-ha-backup --gvc GVC_NAME -o yaml

cpln workload cron start RELEASE_NAME-postgres-ha-backup --gvc GVC_NAME
cpln workload cron get RELEASE_NAME-postgres-ha-backup --gvc GVC_NAME

​

Restoring a Backup

export PGPASSWORD="PASSWORD"

aws s3 cp "s3://BUCKET_NAME/PREFIX/BACKUP_FILE.sql.gz" - \
  | gunzip \
  | psql \
      --host=RELEASE_NAME-postgres-ha-proxy.GVC_NAME.cpln.local \
      --port=5432 \
      --username=temporal \
      --dbname=postgres

unset PGPASSWORD

​

Upgrading from 1.0.x

​

Upgrading from 1.1.0

​

Upgrading from 1.2.x

• From 1.2.0, HA MinIO backups only: version 1.2.1 replaced postgresHA.backup.minio.accessKey and postgresHA.backup.minio.secretKey with postgresHA.backup.minio.credentialsSecretName, a dictionary secret you create holding accessKey and secretKey; see Backing Up. The HA and single-instance paths now use the same shape.
• From 1.2.0 or 1.2.1: version 1.3.0 renamed the resource limits so the ceiling is no longer ambiguous next to the floor. Rename these keys in your values; an upgrade carrying an old name is refused at render with resources.cpu was RENAMED to resources.maxCpu (or the matching message for the other three):

​

Upgrading from 1.x

cpln workload cron start RELEASE_NAME-postgres-ha-backup --gvc GVC_NAME
cpln workload cron get RELEASE_NAME-postgres-ha-backup --gvc GVC_NAME

cpln helm upgrade RELEASE_NAME oci://ghcr.io/controlplane-com/templates/temporal \
  --version 2.0.0 --gvc GVC_NAME -f values.yaml \
  --set image=temporalio/server:1.30.8 --set adminTools.image=temporalio/admin-tools:1.30.8

• AWS backups: the HA backup no longer attaches AWS’s ReadOnlyAccess policy. Update your bucket-scoped IAM policy to the full action list under Backing Up before upgrading.
• Temporal 1.32 visibility queries are stricter: a list filter that compares a search attribute with the wrong type, or an empty Text filter, now returns an error instead of results.
• New optional features: archival to S3, which needs Temporal 1.31 or later, so turn it on after the step to 1.31; and browser access to the Web UI from listed IP ranges with ui.allowedCIDRs (see Web UI).

​

Upgrading from 2.0.0

cpln secret reveal SECRET_NAME -o yaml   # the 2.0.0 secret: its CURRENT values, until the upgrade deletes it
cpln secret create-dictionary --name RELEASE_NAME-postgres-credentials \
  --entry username='YOUR_CURRENT_USERNAME' --entry password='YOUR_CURRENT_PASSWORD' --entry database=temporal

​

Upgrading from 2.1.0 or earlier

​

Rotating the Database Password

cpln secret reveal SECRET_NAME -o yaml > secret.yaml
# edit only the key you are rotating in secret.yaml
cpln apply -f secret.yaml
cpln secret reveal SECRET_NAME -o yaml

1. Change it in the database: ALTER ROLE DB_USERNAME PASSWORD 'NEW_PASSWORD'; (connected as that user, from inside the GVC).
2. Update the secret as above.
3. Right away, redeploy Temporal, and wait until get-deployments shows a higher version number, alone and ready (right after the command the status may not show the restart yet, and ready: true alone can still describe the old replica). Do not redeploy the database: PostgreSQL reads the secret only when it is first initialized, so restarting it only adds downtime.

cpln workload force-redeployment RELEASE_NAME-temporal --gvc GVC_NAME

pgx() { cpln workload exec RELEASE_NAME-postgres-ha --gvc GVC_NAME --location LOCATION \
  --container patroni-postgres --replica "RELEASE_NAME-postgres-ha-$1" -- "${@:2}"; }
# 1. Find the leader ("role": "leader"); the other two must be "streaming" on the same "timeline"
pgx 0 curl -s localhost:8008/cluster
# 2. If the leader is not RELEASE_NAME-postgres-ha-0, move it there (N = current leader), then repeat step 1
pgx 0 curl -s -XPOST -d '{"leader":"RELEASE_NAME-postgres-ha-N","candidate":"RELEASE_NAME-postgres-ha-0"}' localhost:8008/switchover
# 3. Back to back: change BOTH roles on -0, put the same password in the secret (export, edit, apply above), restart the members
pgx 0 psql -h 127.0.0.1 -d postgres -c "ALTER ROLE DB_USERNAME PASSWORD 'NEW_PASSWORD'; ALTER ROLE repl PASSWORD 'NEW_PASSWORD';"
cpln workload force-redeployment RELEASE_NAME-postgres-ha --gvc GVC_NAME
# 4. Once the members are back, confirm one leader and two "streaming" replicas on one timeline (repeat step 1)
# 5. Optional: remove the install-time password Patroni keeps in its cluster config
pgx 0 curl -s -XPATCH -d '{"postgresql":{"authentication":null}}' localhost:8008/config

pgx N curl -s -XPOST -d '{"force": true}' localhost:8008/reinitialize

​

Scaling and Availability

• replicas sets the server count, from 1 to 5. Every replica runs the full server and they form one Temporal cluster. 3 survives the loss of a replica with capacity to spare; 1 is fine when a short gap per upgrade or restart is acceptable.
• Add replicas for availability, not throughput. Each replica’s database connection pools shrink as replicas are added, so that the whole cluster stays within the bundled database’s connection limit.
• Changing replicas restarts every server replica, not just an added or removed one, because the per-replica pool sizes change. Changing internalAccess also restarts every server replica. Members join or drain and work rebalances on its own.
• Change replicas under light load. While replicas with the old and the new pool sizes run side by side, the total connection count can briefly exceed the database’s limit.
• Rolling restarts with more than one replica take several minutes. Replicas are replaced one at a time. A newly started replica may restart its worker service a few times in its first minutes before settling; clients are not affected.
• Losing a replica: the remaining replicas take over its workflows and the platform replaces it. Timers fire on time and nothing is lost; clients may see a few timed-out calls right after the loss.
• Workers scale independently. Run as many worker workloads and replicas as your task queues need.
• Database availability is the HA cluster’s job. In the default postgresHA mode a PostgreSQL leader failure is handled by Patroni and the HAProxy endpoint keeps the same address. In postgres mode there is no failover.
• Durability is in the database, not the server. The server workload has no volume. cpln helm uninstall deletes the database volume sets, taking every workflow history with it.

​

Troubleshooting

cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-temporal"}' --limit 50 --since 10m

​

Important Notes

• Create the database credentials secret before installing, with database set to temporal. A missing secret wedges the deployment with zero log lines; status.versions[].message in cpln workload get-deployments RELEASE_NAME-temporal --gvc GVC_NAME -o yaml names it.
• Editing the credentials secret does not change the database password — run ALTER ROLE first, then force-redeploy Temporal; in HA mode move the leader first (Rotating the Database Password).
• Upgrade Temporal one minor version at a time, and back up through the database backup first. A skipped minor is refused; see Upgrading from 1.x.
• historyShards is fixed at first install. A later change is ignored, so size it before installing.
• The Web UI has no login. List only addresses you trust in ui.allowedCIDRs; ranges broader than /8 (IPv4) or /32 (IPv6) are refused, and a change takes up to a few minutes to apply.
• To stop archiving, set archival.defaultNamespace: false. Never set archival.enabled: false once archival has been used.
• Use the full .cpln.local hostname in every worker and client; short workload names do not resolve.
• Give each release its own database credentials secret. Secret names are organization-wide, so two releases pointed at one secret share a password.
• Keep the GVC to one location. Each location would run its own servers against its own database.
• Uninstall deletes the database volume sets, taking every workflow history with it. Enable backups if the data matters.

​

External References