PostgreSQL Highly Available

Two things to check when upgrading an existing cluster.Confirm you have a base backup. WAL archiving on its own cannot restore a cluster — it replays from a base backup. If you run wal-g backups, list what is actually in your bucket before relying on them:

cpln workload exec RELEASE_NAME-postgres-ha --gvc GVC_NAME --container wal-g-backup \
  -- wal-g backup-list

Upgrading does not retune a running cluster. Patroni’s consensus timeouts are written to etcd when the cluster is first created, and the chart is never read again afterwards. A new install picks up the current values; an existing cluster keeps the ones it was created with. To move a running cluster, use the patronictl edit-config command in the template README.

Overview

PostgreSQL Highly Available deploys a production-ready PostgreSQL cluster using Patroni for automatic leader election and failover, with etcd providing distributed consensus. An optional HAProxy workload routes all write traffic to the current primary replica, providing a stable connection endpoint regardless of which replica holds the leader role.

For production use, maintain at least 3 PostgreSQL replicas and 3 etcd replicas. etcd requires an odd number of replicas (3, 5, 7) for quorum.

What Gets Created

Stateful Patroni PostgreSQL Workload — A Patroni-managed PostgreSQL cluster with configurable replica count and resources. Each replica has its own volume. Leadership is gracefully handed off before any replica is shut down, ensuring write availability is maintained during deployments and restarts.
etcd Workload — A dedicated etcd cluster providing distributed consensus for Patroni leader election.
HAProxy Leader-Routing Workload (optional, enabled by default) — Routes write traffic to the current primary replica, providing a stable connection endpoint during failover.
PgBouncer Workload (optional) — A connection pooler deployed in front of HAProxy, multiplexing application connections into a smaller pool of real database connections.
Volume Set — Persistent storage for PostgreSQL data, with optional autoscaling.
Secrets — A dictionary secret with database credentials; opaque secrets for the Patroni startup script, HAProxy startup script, and WAL-G backup script (created as needed).
Identity & Policy — An identity bound to the workload with reveal access to all required secrets, and cloud storage access when backup is enabled.

This template does not create a GVC. You must deploy it into an existing GVC.

Prerequisites

Version 2.5.0 is a breaking change. The postgres block was removed: postgres.username, postgres.password and postgres.database are now a prerequisite secret. An install or upgrade that still sets any of them fails at render, before anything is applied, and the error names the replacement. If you are running 2.4.x, read Upgrading From 2.4.x first.

One dictionary secret must exist before you install. These are the credentials your applications put in their connection strings, so they are not values — putting them in values would land them in the Helm release.

cpln secret create-dictionary --name my-postgres-ha-credentials \
  --entry username=myuser \
  --entry password='YOUR-STRONG-PASSWORD' \
  --entry database=mydb

Set config.credentialsSecretName to the name you used. Secret names are organization-wide, so give each release its own.

If the secret does not exist at install time, the deployment wedges silently. cpln logs returns zero lines — the container never starts, so it has nothing to log. The one place the reason appears is status.versions[].message:

cpln workload get-deployments RELEASE_NAME-postgres-ha --gvc GVC_NAME -o yaml

Note this is get-deployments — plain cpln workload get has no versions field. Creating the secret repairs the deployment on its own in roughly 5.5 to 10.5 minutes, or force a redeployment to skip the wait.

Enabling backups to MinIO needs a second dictionary secret — see Backup Prerequisites.

Upgrading From 2.4.x

Delete the postgres: block from your values, create the credentials secret above, and set config.credentialsSecretName.

Use the credentials your cluster already has. They were applied when the data directory was first initialised and are still what PostgreSQL enforces. Putting new values in the secret does not change the running database — it just gives your applications a password that no longer works.

Versions 2.4.2 and earlier shipped username: username and password: password as defaults. If your cluster is still running those, treat them as compromised: they are published in a public repository. Change them inside PostgreSQL with ALTER ROLE ... WITH PASSWORD and put the new value in the secret.

Installation

To install, follow the instructions for your preferred method:

UI

Browse, install, and manage templates visually

CLI

Manage templates from your terminal

Terraform

Declare templates in your Terraform configurations

Pulumi

Declare templates in your Pulumi programs

Configuration

The default values.yaml for this template:

replicas: 3

resources:
  minCpu: 500m
  minMemory: 1Gi
  maxCpu: 1
  maxMemory: 2Gi

image: controlplanecorporation/patroni-postgres:0.7

config:
  # REQUIRED prerequisite `dictionary` secret holding `username`, `password`
  # and `database`. It must EXIST BEFORE INSTALL.
  credentialsSecretName: my-postgres-ha-credentials

multiZone: false

volumeset:
  capacity: 10 # initial capacity in GiB (minimum is 10)
  autoscaling:
    enabled: false
    maxCapacity: 100 # Maximum capacity in GiB
    minFreePercentage: 10 # Trigger scaling when free space drops below this percentage
    scalingFactor: 1.2 # Multiply current capacity by this factor when scaling up

internal_access:
  type: same-gvc # options: same-gvc, same-org, workload-list
  workloads: # Note: can only be used if type is same-gvc or workload-list
    #- //gvc/GVC_NAME/workload/WORKLOAD_NAME

etcd:
  replicas: 3
  resources:
    cpu: 500m
    memory: 512Mi
  multiZone: false
  tuning:
    # Passed through to the bundled etcd. Without compaction its backend grows
    # with time alone — Patroni renews its lease every ~10s and every renewal is
    # a revision — until it hits the quota and etcd goes READ-ONLY.
    autoCompactionMode: periodic # periodic (retention is a duration) or revision (a revision count)
    autoCompactionRetention: 1h # periodic needs an explicit unit (1h, 30m, 24h); revision takes a count
    quotaBackendBytes: 0 # backend size limit in bytes; 0 = etcd's own default of 2 GiB
  volumeset:
    capacity: 10 # initial capacity in GiB (minimum is 10)
  internal_access:
    type: same-gvc # options: same-gvc, same-org, workload-list
    workloads:
      #- //gvc/GVC_NAME/workload/WORKLOAD_NAME

pgbouncer:
  enabled: false
  image: edoburu/pgbouncer:v1.25.1-p0
  poolMode: transaction  # options: session, transaction, statement
  defaultPoolSize: 25    # number of real Postgres connections PgBouncer maintains per pod
  maxClientConn: 1000    # maximum number of client connections PgBouncer accepts per pod
  maxDbConnections: 100  # hard cap on total Postgres connections regardless of how many PgBouncer pods are running
  minReplicas: 2
  maxReplicas: 4

  resources:
    cpu: 200m
    memory: 128Mi

proxy: # HAProxy endpoint to write to leader replica. Automatically enabled when pgbouncer is enabled.
  enabled: true
  image: haproxy:2.9
  resources:
    cpu: 100m
    memory: 128Mi
  minReplicas: 2
  maxReplicas: 2

backup:
  enabled: false
  mode: logical  # logical or wal-g
  resources:
    cpu: 100m
    memory: 128Mi

  logical:
    image: ghcr.io/controlplane-com/backup-images/postgres-backup:17.1.0 # tag 17.1.0 = Postgres 17, 18.1.0 = Postgres 18
    schedule: "0 2 * * *" # cron schedule, default is daily at 2am UTC

  walg:
    intervalSeconds: 21600 # interval in seconds between backups, default is every 6 hours

  # storage settings are applied to whichever mode is enabled
  provider: aws # Options: aws, gcp, or minio

  aws:
    bucket: pg-ha-backup-bucket
    region: us-east-1
    cloudAccountName: my-s3-cloud-account
    policyName: pg-ha-backup-policy
    prefix: postgres/backups

  gcp:
    bucket: pg-ha-backup-bucket
    cloudAccountName: my-gcs-cloud-account
    prefix: postgres/backups

  minio: # Backup to a self-hosted MinIO workload (or any S3-compatible endpoint)
    endpoint: http://my-minio-workload:9000 # e.g. http://WORKLOAD_NAME:9000 for an internal MinIO template deployment
    bucket: pg-ha-backup-bucket
    # REQUIRED prerequisite `dictionary` secret holding `accessKey` and
    # `secretKey`, only when backups are enabled with provider: minio
    credentialsSecretName: my-postgres-ha-minio-credentials
    prefix: postgres/backups

Credentials

config.credentialsSecretName — Name of the dictionary secret holding username, password and database. PostgreSQL creates that user and that database on first boot, and this is the credential your applications put in their connection strings.

The secret must exist before you install — see Prerequisites. The cluster reads it through cpln://secret/... references, so the values appear in neither the Helm release nor the stored workload spec.

These credentials are only applied on first startup, when the data directory is empty. Rotating the secret afterwards does not change the stored password; change it inside PostgreSQL with ALTER ROLE ... WITH PASSWORD and update the secret to match.

PostgreSQL Cluster

replicas — Number of PostgreSQL replicas. Minimum 3 recommended for production.
resources.minCpu / resources.minMemory — Minimum CPU and memory guaranteed per replica.
resources.maxCpu / resources.maxMemory — Maximum CPU and memory per replica.
multiZone — Spread replicas across availability zones within the location.

Storage

volumeset.capacity — Initial volume size in GiB (minimum 10). Each replica gets its own volume.
volumeset.autoscaling.enabled — Automatically expand the volume as it fills. When enabled:
- maxCapacity — Maximum volume size in GiB.
- minFreePercentage — Trigger a scale-up when free space drops below this percentage.
- scalingFactor — Multiply the current capacity by this factor when scaling up.

etcd Cluster

etcd.replicas — Number of etcd replicas. Must be an odd number (3, 5, 7) for quorum.
etcd.resources.cpu / etcd.resources.memory — CPU and memory per etcd replica.
etcd.multiZone — Spread etcd replicas across availability zones.
etcd.volumeset.capacity — Initial volume size for etcd data in GiB.
etcd.internal_access.type — Controls which workloads can reach the etcd cluster.

etcd History Compaction

etcd.tuning.autoCompactionMode, etcd.tuning.autoCompactionRetention and etcd.tuning.quotaBackendBytes control how much revision history the bundled etcd cluster keeps and how large its backend may grow. The defaults — periodic, 1h and 0 (etcd’s own 2 GiB limit) — are the right settings for a Patroni consensus store and should be left alone. The etcd template documents the mechanism, the accepted value formats, and the read-only commands for inspecting a cluster.

Template versions before 2.4.2 did not compact the bundled etcd cluster. Patroni renews its leader lease about every 10 seconds and every renewal creates a revision, so etcd grows with time alone — roughly 19 MB per day on an idle cluster — and reaches its 2 GiB backend quota in about 110 days. etcd then goes read-only across the whole cluster, and the PostgreSQL replicas restart-loop with exitCode: 0 and reason: Completed, which reads as healthy and is easily mistaken for a database fault. Upgrade to 2.4.2 or later to turn compaction on.Upgrading prevents further growth but cannot shrink a backend that has already grown, and a cluster that has already raised a NOSPACE alarm needs operator recovery rather than an upgrade — see If the Backend Quota Is Already Full.

HAProxy (Strongly Recommended)

In a Patroni cluster, only the leader replica accepts writes — other replicas are read-only. HAProxy automatically routes write traffic to the current leader, providing a stable connection endpoint even during failover. HAProxy exposes two HTTP endpoints on the proxy workload for observability:

Endpoint	Description
`:8404/healthz`	Returns healthy when at least one primary backend is reachable, unhealthy otherwise
`:8405/stats`	Live stats page showing connection counts and the health status of each replica

proxy.enabled — Deploy the HAProxy leader-routing workload (default: true).
proxy.resources.cpu / proxy.resources.memory — CPU and memory per HAProxy replica.
proxy.minReplicas / proxy.maxReplicas — Replica count for the proxy workload.

HAProxy must be enabled (proxy.enabled: true) for logical backups to function correctly. WAL-G backups do not require the proxy.

Internal Access

internal_access.type — Controls which workloads can connect to PostgreSQL on port 5432:

Type	Description
`same-gvc`	Allow access from all workloads in the same GVC
`same-org`	Allow access from all workloads in the same organization
`workload-list`	Allow access only from specific workloads listed in `workloads`

PgBouncer Connection Pooling

PgBouncer is an optional connection pooler that sits in front of HAProxy, multiplexing application connections into a smaller pool of real database connections. HAProxy handles leader routing and failover transparently — PgBouncer is unaware of which PostgreSQL replica is the primary. Enabling PgBouncer automatically enables HAProxy, as it is required for leader-aware routing in the HA cluster. When enabled, PgBouncer becomes the primary connection endpoint:

RELEASE_NAME-pgbouncer.GVC_NAME.cpln.local:5432

pgbouncer.poolMode — Controls how connections are reused:

Mode	Description
`transaction`	Connection held only for the duration of a transaction, then returned to the pool. Best for most web and API workloads. Not compatible with session-level features (`SET` variables, temporary tables, advisory locks).
`session`	Connection held for the entire client session. Compatible with all PostgreSQL features but provides less reuse. Increase `defaultPoolSize` to match your expected concurrent client count.
`statement`	Connection returned after every statement. Transactions are not supported. Rarely used.

pgbouncer.defaultPoolSize — Number of real PostgreSQL connections PgBouncer maintains per pod (default: 25).
pgbouncer.maxClientConn — Maximum number of client connections PgBouncer accepts per pod (default: 1000).
pgbouncer.maxDbConnections — Hard cap on total PostgreSQL connections across all PgBouncer pods. Prevents connection blowout when PgBouncer scales out — set this to a value the PostgreSQL primary can safely handle (default: 100).
pgbouncer.minReplicas / pgbouncer.maxReplicas — PgBouncer autoscales on RPS between these bounds. Increase maxReplicas for high-throughput workloads.
pgbouncer.resources.cpu / pgbouncer.resources.memory — Resources allocated to each PgBouncer pod.

Application Retry Logic

Your application should implement retry logic on database connections. Several normal cluster events surface transient errors to the client:

Patroni failover — During a leader election, the old primary steps down and a new one is promoted. HAProxy detects the change and re-routes writes, but connections in flight during the transition may receive a connection reset or brief refusal.
Rolling restarts and deployments — Each replica is restarted in turn. Leadership is handed off gracefully before the leader shuts down, but applications may see a momentary connection disruption while the new leader is established and HAProxy updates its routing.
Scale down — Removing replicas triggers a shutdown sequence. If the departing replica held the leader role, a new election occurs and writes are unavailable until the new leader is ready.

These events are expected and short-lived. Implementing exponential backoff with retry on connection errors ensures your application recovers automatically without surfacing errors to end users.

Connecting to PostgreSQL

Connect to PostgreSQL through the HAProxy workload, which always routes to the current leader:

RELEASE_NAME-postgres-ha-proxy.GVC_NAME.cpln.local:5432

Backup

Two backup modes are available. Set backup.enabled: true, choose a mode, and configure the storage provider.

Mode	Description
`logical`	Scheduled SQL dumps via `pg_dump`. Portable and suitable for smaller databases. Requires HAProxy to be enabled.
`wal-g`	Continuous WAL archiving with point-in-time recovery. Suitable for larger databases requiring minimal data loss. Runs as a sidecar container alongside PostgreSQL.

backup.mode — logical or wal-g.
backup.provider — aws, gcp, or minio.
backup.resources.cpu / backup.resources.memory — Resources allocated to the backup container.

Logical backup settings:

backup.logical.schedule — Cron expression for backup frequency (default: daily at 2am UTC).

WAL-G backup settings:

backup.walg.intervalSeconds — Interval between base backups in seconds (default: 21600, every 6 hours).

Cluster Metrics

From version 2.7.0, Patroni’s Prometheus metrics are scraped into Control Plane’s built-in metrics stack automatically. There is no exporter to run and nothing to configure — they appear in Grafana alongside the standard workload metrics, labelled by workload, replica, and container.

Metric	Tells you
`patroni_primary`	Which replica currently holds the leader lock — `1` on exactly one member.
`patroni_replica`	Which members are followers.
`patroni_postgres_streaming`	Whether a follower is actually streaming from the primary.
`patroni_xlog_location`	WAL position per member — the basis for replication lag.
`patroni_postgres_timeline`	Timeline number; increments on every failover.
`patroni_dcs_last_seen`	When this member last reached etcd.
`patroni_failsafe_mode_is_active`	`1` while the primary is riding out an etcd outage instead of demoting.
`patroni_cluster_unlocked`	`1` when no member holds the leader lock.

Worth alerting on: patroni_primary summing to anything other than 1 across the cluster, patroni_cluster_unlocked staying 1, or patroni_postgres_streaming dropping to 0 on a follower.

These are cluster-state metrics, not database performance. They answer who is primary, whether replication is streaming, and whether a failover happened — not connection counts, cache hit ratio, or slow queries. Those would require a postgres_exporter sidecar, which this template does not include.

You can also query them directly from the org’s Prometheus federation endpoint:

curl -H "Authorization: Bearer $(cpln profile token)" \
  --get --data-urlencode 'match[]=patroni_primary' \
  'https://metrics.cpln.io/metrics/org/ORG_NAME/api/v1/federate'

Backup Prerequisites

AWS S3

Before enabling backup with provider: aws, complete the following in your AWS account:

Create an S3 bucket. Set backup.aws.bucket to the bucket name and backup.aws.region to its region.
If you do not have a Cloud Account set up, refer to the docs to Create a Cloud Account. Set backup.aws.cloudAccountName to its name.
Create an IAM policy with the following JSON, replacing YOUR_BUCKET_NAME:

Version 2.5.1 narrows AWS backup permissions. This version removes aws::ReadOnlyAccess from the backup identity. That AWS managed policy granted read access to every bucket in your AWS account and contains no write actions at all, so it was never carrying the backup itself — but it was silently supplying any read action your own bucket-scoped policy happened to omit.Update your IAM policy to the full action list below before upgrading. If it already matches, no action is needed. The identity now carries cpln-connector and your bucket-scoped policy only, which is strictly narrower than before. Nothing else changes.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:ListBucket",
                "s3:GetObjectVersion",
                "s3:DeleteObjectVersion",
                "s3:GetBucketLocation",
                "s3:AbortMultipartUpload",
                "s3:ListBucketMultipartUploads",
                "s3:ListMultipartUploadParts"
            ],
            "Resource": [
                "arn:aws:s3:::YOUR_BUCKET_NAME",
                "arn:aws:s3:::YOUR_BUCKET_NAME/*"
            ]
        }
    ]
}

Set backup.aws.policyName to the name of the policy created in step 3.
Set backup.aws.prefix to the folder path where backups will be stored.

GCS

Before enabling backup with provider: gcp, complete the following in your GCP account:

Create a GCS bucket. Set backup.gcp.bucket to the bucket name.
If you do not have a Cloud Account set up, refer to the docs to Create a Cloud Account. Set backup.gcp.cloudAccountName to its name.
Add the Storage Admin role to the GCP service account associated with the Cloud Account.
Set backup.gcp.prefix to the folder path where backups will be stored.

MinIO

Before enabling backup with provider: minio, ensure your MinIO instance is accessible:

Create a bucket in MinIO. Set backup.minio.bucket to its name.
Set backup.minio.endpoint to the MinIO S3 API address including the port. For the minio marketplace template deployed in the same GVC, use http://WORKLOAD_NAME:9000.
Create a dictionary secret holding exactly the keys accessKey and secretKey, and set backup.minio.credentialsSecretName to its name. For the minio template these are its admin.username and admin.password:

cpln secret create-dictionary --name my-postgres-ha-minio-credentials \
  --entry accessKey=MINIO_ACCESS_KEY \
  --entry secretKey=MINIO_SECRET_KEY

Set backup.minio.prefix to the folder path where backups will be stored.

MinIO backup requires no Control Plane Cloud Account — credentials are passed directly. Both logical and wal-g modes are supported; WAL-G uses its native S3-compatible storage support (AWS_ENDPOINT + path-style addressing).

Restoring a Backup

Logical

Run the following from a client with access to the backup bucket. Connect through the proxy workload so the restore targets the current leader. AWS S3:

export PGPASSWORD="PASSWORD"

aws s3 cp "s3://BUCKET_NAME/PREFIX/BACKUP_FILE.sql.gz" - \
  | gunzip \
  | psql \
      --host=RELEASE_NAME-postgres-ha-proxy.GVC_NAME.cpln.local \
      --port=5432 \
      --username=USERNAME \
      --dbname=postgres

unset PGPASSWORD

GCS:

export PGPASSWORD="PASSWORD"

gsutil cp "gs://BUCKET_NAME/PREFIX/BACKUP_FILE.sql.gz" - \
  | gunzip \
  | psql \
      --host=RELEASE_NAME-postgres-ha-proxy.GVC_NAME.cpln.local \
      --port=5432 \
      --username=USERNAME \
      --dbname=postgres

unset PGPASSWORD

MinIO:

export PGPASSWORD="PASSWORD"
export AWS_ACCESS_KEY_ID="MINIO_ACCESS_KEY"
export AWS_SECRET_ACCESS_KEY="MINIO_SECRET_KEY"
aws configure set default.s3.addressing_style path

aws s3 cp "s3://BUCKET_NAME/PREFIX/BACKUP_FILE.sql.gz" - --endpoint-url "http://MINIO_ENDPOINT:9000" \
  | gunzip \
  | psql \
      --host=RELEASE_NAME-postgres-ha-proxy.GVC_NAME.cpln.local \
      --port=5432 \
      --username=USERNAME \
      --dbname=postgres

unset PGPASSWORD AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY

WAL-G

WAL-G point-in-time restore requires an empty data directory. Follow these steps:

Run wal-g backup-list to identify the desired backup.
Stop the PostgreSQL workload.
Create a new Volume Set for the restored data.
Run a one-off restore workload with the new Volume Set mounted at /var/lib/postgresql/data and run:

wal-g backup-fetch /var/lib/postgresql/data/pgdata <backup_name>

Re-point the PostgreSQL workload to the restored Volume Set and restart.
After the restore, change backup.walg.prefix before re-enabling backups to avoid system identifier conflicts.

External References

Patroni Documentation

Patroni automatic failover documentation

PostgreSQL Documentation

Official PostgreSQL documentation

etcd Documentation

Official etcd documentation

PgBouncer Documentation

PgBouncer configuration reference

Backup Image Source

Source code for the PostgreSQL backup container image

PostgreSQL HA Template

View the source files, default values, and chart definition

Template Catalog

Templates

PostgreSQL Highly Available

Overview

What Gets Created

Prerequisites

Upgrading From 2.4.x

Installation

UI

CLI

Terraform

Pulumi

Configuration

Credentials

PostgreSQL Cluster

Storage

etcd Cluster

etcd History Compaction

HAProxy (Strongly Recommended)

Internal Access

PgBouncer Connection Pooling

Application Retry Logic

Connecting to PostgreSQL

Backup

Cluster Metrics

Backup Prerequisites

AWS S3

GCS

MinIO

Restoring a Backup

Logical

WAL-G

External References

Patroni Documentation

PostgreSQL Documentation

etcd Documentation

PgBouncer Documentation

Backup Image Source

PostgreSQL HA Template

​Overview

​What Gets Created

​Prerequisites

​Upgrading From 2.4.x

​Installation

UI

CLI

Terraform

Pulumi

​Configuration

​Credentials

​PostgreSQL Cluster

​Storage

​etcd Cluster

​etcd History Compaction

​HAProxy (Strongly Recommended)

​Internal Access

​PgBouncer Connection Pooling

​Application Retry Logic

​Connecting to PostgreSQL

​Backup

​Cluster Metrics

​Backup Prerequisites

​AWS S3

​GCS

​MinIO

​Restoring a Backup

​Logical

​WAL-G

​External References

Patroni Documentation

PostgreSQL Documentation

etcd Documentation

PgBouncer Documentation

Backup Image Source

PostgreSQL HA Template

Overview

What Gets Created

Prerequisites

Upgrading From 2.4.x

Installation

Configuration

Credentials

PostgreSQL Cluster

Storage

etcd Cluster

etcd History Compaction

HAProxy (Strongly Recommended)

Internal Access

PgBouncer Connection Pooling

Application Retry Logic

Connecting to PostgreSQL

Backup

Cluster Metrics

Backup Prerequisites

AWS S3

GCS

MinIO

Restoring a Backup

Logical

WAL-G

External References