PostgreSQL Highly Available

Overview

PostgreSQL Highly Available deploys a multi-member PostgreSQL 17 cluster in one location. Patroni manages the members, elects a leader and fails over automatically when the leader is lost; a dedicated etcd cluster holds the consensus state. An HAProxy workload gives applications one stable address that always routes to the current leader, and an optional PgBouncer in front of it pools connections. Backups are optional, in two modes: scheduled pg_dumpall dumps, or continuous WAL archiving with WAL-G. Database credentials are not template values. The cluster reads its username, password and database name from a dictionary secret you create before installing, so no password passes through Helm or lands in the release. The template creates no credential secret of its own.

This template deploys into an existing GVC that you already have. It does not create or manage a GVC. Every workload runs in each location the GVC has, so use a GVC with a single location for this template.

What Gets Created

Resource	Name	Purpose	Optional
Workload	`RELEASE_NAME-postgres-ha`	Patroni-managed PostgreSQL 17 members (stateful, `replicas` replicas, TCP `5432`, Patroni REST API `8008`); container `patroni-postgres`	—
Workload	`RELEASE_NAME-etcd`	Consensus store for leader election (stateful, `etcd.replicas` replicas, TCP `2379`/`2380`); container `etcd`	—
Workload	`RELEASE_NAME-postgres-ha-proxy`	HAProxy routing TCP `5432` to the current leader; health on `8404`, stats on `8405`; container `haproxy`	`proxy.enabled: true` (default), forced on by `pgbouncer.enabled: true`
Workload	`RELEASE_NAME-pgbouncer`	PgBouncer connection pooler in front of HAProxy (TCP `5432`); container `pgbouncer`	`pgbouncer.enabled: true`
Workload	`RELEASE_NAME-postgres-ha-backup`	Cron job running `pg_dumpall` through HAProxy to S3, GCS or MinIO; container `backup`	`backup.enabled: true` with `backup.mode: logical`
Container	`wal-g-backup` on `RELEASE_NAME-postgres-ha`	Sidecar that takes a WAL-G base backup from the leader every `backup.walg.intervalSeconds`	`backup.enabled: true` with `backup.mode: wal-g`
Volume Set	`RELEASE_NAME-postgres-ha-vs`	One `ext4` general-purpose SSD volume per member holding `PGDATA`; a final snapshot is taken when the volume set is deleted and kept 7 days	—
Volume Set	`RELEASE_NAME-etcd-vs`	One volume per etcd member	—
Identity	`RELEASE_NAME-postgres-ha-identity`	Shared by the members, proxy, pooler and backup workloads; carries the cloud account binding when backups use AWS or GCP	—
Policy	`RELEASE_NAME-postgres-ha-policy`	Grants the identity `reveal` on the chart’s startup-script secrets and on exactly the credential secrets you created	—
Identity, Policy	`RELEASE_NAME-etcd-identity`, `RELEASE_NAME-etcd-policy`	The etcd cluster’s own identity and `reveal` grant on its startup script	—
Secret	`RELEASE_NAME-postgres-proxy-startup`	Patroni startup script, mounted into every member	—
Secret	`RELEASE_NAME-patroni-startup`	HAProxy startup script	`proxy.enabled: true`
Secret	`RELEASE_NAME-wal-g-backup-script`	WAL-G sidecar script	—
Secret	`RELEASE_NAME-etcd-startup`	etcd startup script	—
Secret	`RELEASE_NAME-postgres-config`	Non-sensitive backup settings (bucket, region, MinIO endpoint)	`backup.enabled: true`

Prerequisites

One secret must exist before you install. It holds the credentials your applications put in their connection strings. Secrets are org-level, so no GVC flag is involved.

Create the database credentials secret

A dictionary secret holding exactly three keys — username, password and database. Patroni creates that user as the cluster superuser and that database the first time the cluster is initialized, and uses the same password for its internal replication role:

cpln secret create-dictionary --name CREDENTIALS_SECRET_NAME \
  --entry username=postgres \
  --entry password='YOUR-STRONG-PASSWORD' \
  --entry database=mydb

Set config.credentialsSecretName to this name. Read it back later with cpln secret reveal CREDENTIALS_SECRET_NAME -o yaml — a bare cpln secret reveal prints only a summary table, not the values. Secret names are org-wide, so give each release its own.

Create the secret before installing, or the deployment wedges silently. A config.credentialsSecretName that points at a secret which does not exist installs successfully and then never starts. The container never runs, so cpln logs returns zero lines. The one place the reason appears is status.versions[].message:

cpln workload get-deployments RELEASE_NAME-postgres-ha --gvc GVC_NAME -o yaml

The secret CREDENTIALS_SECRET_NAME no longer exists. Workload updates are paused until the
secret is added or the reference to the secret removed.

Use get-deployments — plain cpln workload get has no versions key. Creating the missing secret repairs the deployment on its own within several minutes, or run cpln workload force-redeployment RELEASE_NAME-postgres-ha --gvc GVC_NAME to skip the wait.

Backups are optional and need a bucket plus, for AWS or GCP, a Control Plane cloud account; MinIO backups need a second dictionary secret holding accessKey and secretKey — see Backing Up. Nothing else is required for a default install.

Installation

Install the chart from the marketplace registry, pointing it at the secret you created:

cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/postgres-highly-available \
  --version 2.7.0 --gvc GVC_NAME \
  --set config.credentialsSecretName=CREDENTIALS_SECRET_NAME

Or follow the instructions for your preferred method:

UI

Browse, install, and manage templates visually

CLI

Manage templates from your terminal

Terraform

Declare templates in your Terraform configurations

Pulumi

Declare templates in your Pulumi programs

Configuration

Each block below is the shipped default for that part of values.yaml.

Cluster and Resources

replicas: 3

resources:
  minCpu: 500m
  minMemory: 1Gi
  maxCpu: 1
  maxMemory: 2Gi

# Patroni's Prometheus metrics are scraped by the platform automatically on the
# REST API port — no knob, no exporter. See "Cluster metrics" in the README.
image: controlplanecorporation/patroni-postgres:0.7

multiZone: false

replicas — Number of PostgreSQL members. Keep at least 3 for a cluster that survives the loss of one member; one member is the leader and accepts writes, the others stream from it.
multiZone — Spread the members across availability zones within the location, when the location supports it.

On a stateful workload the ratio of maxCpu to minCpu may not exceed 4:1. Raising maxCpu without raising minCpu can cross that limit and is rejected when the workload is applied.

Credentials

config:
  # REQUIRED PREREQUISITE SECRET — CREATE IT BEFORE YOU INSTALL.
  # A `dictionary` secret holding exactly three keys: `username`, `password` and
  # `database`. If it does not exist at install time the deployment WEDGES
  # silently — `cpln logs` returns nothing at all. See Prerequisites in the
  # README for the exact `cpln secret create-dictionary` command.
  credentialsSecretName: my-postgres-ha-credentials

The three keys are applied when the cluster is first initialized. Changing the secret afterwards does not change the stored passwords — see Rotating the Password.

Storage

volumeset:
  capacity: 10 # initial capacity in GiB (minimum is 10)
  autoscaling:
    enabled: false # Set to true to enable autoscaling
    maxCapacity: 100 # Maximum capacity in GiB when autoscaling is enabled
    minFreePercentage: 10 # Minimum free percentage to trigger scaling when autoscaling is enabled
    scalingFactor: 1.2 # Scaling factor to determine how much to scale up when autoscaling is triggered

Each member has its own volume of this size. The cluster caps the WAL retained for a member that is offline at a quarter of volumeset.capacity, so a member that stays down does not fill the leader’s disk; a member that exceeds the cap is re-cloned from the leader when it returns.

Internal Access

internal_access:
  type: same-gvc # options: same-gvc, same-org, workload-list
  workloads:  # Note: can only be used if type is same-gvc or workload-list
    #- //gvc/GVC_NAME/workload/WORKLOAD_NAME
    #- //gvc/GVC_NAME/workload/WORKLOAD_NAME

Type	Who can reach port `5432`
`same-gvc`	Every workload in the same GVC
`same-org`	Every workload in the same org
`workload-list`	Only the workloads listed in `internal_access.workloads`

The same setting is applied to the members, the HAProxy workload and the PgBouncer workload. A change can take several minutes to take effect — re-test before concluding it did not apply.

With workload-list, the list must also contain the release’s own workloads, because the members replicate from each other and HAProxy, PgBouncer and the backup job all connect to them: //gvc/GVC_NAME/workload/RELEASE_NAME-postgres-ha, //gvc/GVC_NAME/workload/RELEASE_NAME-postgres-ha-proxy, and RELEASE_NAME-pgbouncer and RELEASE_NAME-postgres-ha-backup when they are enabled — plus your applications. A list without them leaves HAProxy waiting for members it cannot reach.

etcd Cluster

The bundled etcd template provides the consensus store. Its own values are passed through under etcd:

etcd:
  replicas: 3
  resources:
    cpu: 500m
    memory: 512Mi
  multiZone: false
  tuning:
    # Passed through to the bundled etcd. Without compaction its backend grows
    # with time alone — Patroni renews its lease every ~10s and every renewal is
    # a revision — until it hits the quota and etcd goes READ-ONLY.
    autoCompactionMode: periodic # periodic (retention is a duration) or revision (a revision count)
    autoCompactionRetention: 1h # periodic needs an explicit unit (1h, 30m, 24h); revision takes a count
    quotaBackendBytes: 0 # backend size limit in bytes; 0 = etcd's own default of 2 GiB
  volumeset:
    capacity: 10 # initial capacity in GiB (minimum is 10)
  internal_access:
    type: same-gvc # options: same-gvc, same-org, workload-list
    workloads:  # Note: can only be used if type is same-gvc or workload-list
      #- //gvc/GVC_NAME/workload/WORKLOAD_NAME
      #- //gvc/GVC_NAME/workload/WORKLOAD_NAME

etcd.replicas — Must be an odd number of at least 3 (3, 5, 7); the chart refuses other values at render. Choose it at install time — the etcd template does not support changing the member count on a running release.
etcd.internal_access — Must allow the PostgreSQL members to reach etcd on port 2379. With workload-list, include //gvc/GVC_NAME/workload/RELEASE_NAME-postgres-ha and //gvc/GVC_NAME/workload/RELEASE_NAME-etcd.

etcd History Compaction

Patroni renews its leader lease every few seconds and every renewal is a new etcd revision, so the etcd backend grows with time alone unless old revisions are compacted. Uncompacted, an otherwise idle cluster reaches etcd’s 2 GiB backend quota in a matter of months, etcd raises a NOSPACE alarm and goes read-only, and the PostgreSQL members can no longer renew their leases. The etcd.tuning defaults — periodic, 1h and 0 (etcd’s own 2 GiB limit) — keep the backend flat and should be left alone. The etcd template documents the mechanism, the accepted value formats and the read-only inspection commands. Template versions before 2.4.2 did not compact — see Upgrading From a Version Before 2.4.2.

HAProxy Leader Endpoint

Only the leader accepts writes; the other members are read-only. HAProxy checks every member’s Patroni REST API and sends port 5432 traffic only to the one that reports itself primary, so applications keep one address through a failover.

proxy: # HA Proxy endpoint to write to leader replica. Automatically enabled when pgbouncer is enabled.
  enabled: true
  image: haproxy:2.9
  resources:
    cpu: 100m
    memory: 128Mi
  minReplicas: 2
  maxReplicas: 2

Endpoint on `RELEASE_NAME-postgres-ha-proxy`	Purpose
`:5432`	PostgreSQL, routed to the current leader
`:8404/healthz`	Returns healthy while at least one member reports itself primary
`:8405/stats`	HAProxy stats page showing each member’s health-check state

HAProxy must be enabled for logical backups, which connect through it. It is forced on when pgbouncer.enabled: true. HAProxy reads the member list from replicas when it starts, so after changing replicas force a redeployment of RELEASE_NAME-postgres-ha-proxy.

PgBouncer Connection Pooling

PgBouncer multiplexes many application connections into a smaller pool of real database connections. It connects to HAProxy, so leader routing and failover are handled for it, and it reads the same credentials secret — nothing extra to configure. When enabled it becomes the endpoint your applications connect to.

pgbouncer:
  enabled: false
  image: edoburu/pgbouncer:v1.25.1-p0
  poolMode: transaction  # options: session, transaction, statement
  defaultPoolSize: 25    # number of real Postgres connections PgBouncer maintains per pod
  maxClientConn: 1000    # maximum number of client connections PgBouncer accepts per pod
  maxDbConnections: 100  # hard cap on total Postgres connections regardless of how many PgBouncer pods are running
  minReplicas: 2
  maxReplicas: 4

  resources:
    cpu: 200m
    memory: 128Mi

`poolMode`	Behaviour
`transaction`	The server connection is held for one transaction. Best for most web and API workloads; not compatible with `SET` variables, temporary tables or advisory locks.
`session`	Held for the whole client session. Compatible with everything, less reuse — raise `defaultPoolSize` to your expected concurrency.
`statement`	Returned after every statement. Transactions are not supported.

maxDbConnections is enforced per PgBouncer replica, not across the deployment. PgBouncer replicas do not coordinate, so the real ceiling on server connections is maxReplicas × maxDbConnections — 4 × 100 = 400 with the shipped values, against a cluster max_connections of 100, of which 3 are reserved for superuser maintenance. Size maxReplicas × maxDbConnections to stay under 97, and treat defaultPoolSize the same way, or clients get remaining connection slots are reserved under load instead of being queued.

PgBouncer autoscales on requests per second between minReplicas and maxReplicas.

Backup

Two modes share the same provider settings. logical runs a cron workload that dumps every database and role with pg_dumpall through HAProxy; wal-g adds a sidecar to each member that archives WAL continuously and takes a base backup from the leader every backup.walg.intervalSeconds. Each provider’s bucket, cloud account and policy steps are under Backing Up.

backup:
  enabled: false
  mode: logical  # logical or wal-g
  resources: # applies to whichever mode is enabled
    cpu: 100m
    memory: 512Mi # 128Mi OOMKills wal-g mid-backup (exitCode 137) and silently produces no base backup

  logical: # logical backup settings
    image: ghcr.io/controlplane-com/backup-images/postgres-backup:17.1.0 # tag 17.1.0 = Postgres 17, 18.1.0 = Postgres 18
    schedule: "0 2 * * *" # cron schedule, default isdaily at 2am UTC

  walg: # wal-g backup settings
    intervalSeconds: 21600 # interval in seconds between backups, default is every 6 hours

  # storage settings are applied to whichever mode is enabled
  provider: aws # Options: aws, gcp, or minio

  aws:
    bucket: pg-ha-backup-bucket
    region: us-east-1
    cloudAccountName: my-s3-cloud-account
    policyName: pg-ha-backup-policy
    prefix: postgres/backups # folder name where your backups will be stored

  gcp:
    bucket: pg-ha-backup-bucket
    cloudAccountName: my-gcs-cloud-account
    prefix: postgres/backups # folder name where your backups will be stored

  minio: # Backup to a self-hosted MinIO workload (or any S3-compatible endpoint)
    endpoint: http://my-minio-workload:9000 # e.g. http://WORKLOAD_NAME:9000 for an internal MinIO template deployment
    bucket: pg-ha-backup-bucket
    # REQUIRED PREREQUISITE SECRET when provider is `minio` — a `dictionary`
    # secret holding `accessKey` and `secretKey`. See Backup setup in the README.
    credentialsSecretName: my-postgres-ha-minio-credentials
    prefix: postgres/backups # folder name where your backups will be stored

Mode	What is written	Restores to
`logical`	One `postgres-TIMESTAMP.sql.gz` object per run of `backup.logical.schedule`	Any PostgreSQL 17 server, by replaying the SQL
`wal-g`	Continuous WAL segments (`archive_timeout` 60 s, zstd) plus a base backup every `backup.walg.intervalSeconds`	An empty data directory, to the latest base backup or a point in time

Connecting

This template exposes no public endpoint. Everything below is reachable from inside the GVC, subject to internal_access.type.

What	Address	Notes
PostgreSQL, writes and reads	`RELEASE_NAME-postgres-ha-proxy.GVC_NAME.cpln.local:5432`	HAProxy routes to the current leader. Use this unless PgBouncer is enabled.
PostgreSQL, pooled	`RELEASE_NAME-pgbouncer.GVC_NAME.cpln.local:5432`	Only when `pgbouncer.enabled: true`. Prefer it when the pooler is on.
Any member, unrouted	`RELEASE_NAME-postgres-ha.GVC_NAME.cpln.local:5432`	Reaches whichever member answers, which may be a read-only replica. For diagnostics only.
HAProxy health and stats	`RELEASE_NAME-postgres-ha-proxy.GVC_NAME.cpln.local:8404/healthz`, `:8405/stats`	HTTP.
Credentials	`cpln secret reveal CREDENTIALS_SECRET_NAME -o yaml`	The `username`, `password` and `database` keys. Never stored in the Helm release.

To verify from your own machine, tunnel to HAProxy and connect with psql:

cpln port-forward RELEASE_NAME-postgres-ha-proxy 5432:5432 --gvc GVC_NAME

psql "postgresql://USERNAME:PASSWORD@127.0.0.1:5432/DATABASE" -c 'select pg_is_in_recovery()'

pg_is_in_recovery() returns f through HAProxy, because it always reaches the leader. The same query on a member directly returns t on a replica:

cpln workload exec RELEASE_NAME-postgres-ha --gvc GVC_NAME --replica RELEASE_NAME-postgres-ha-0 --container patroni-postgres -- psql -h 127.0.0.1 -U USERNAME -d DATABASE -c 'select pg_is_in_recovery()'

Use --replica to pick a member (RELEASE_NAME-postgres-ha-0 through RELEASE_NAME-postgres-ha-N); without it the CLI picks one for you. The superuser credentials are already present in the container’s environment, so psql does not prompt. Applications should retry on connection errors with backoff. A failover, a rolling restart or a scale-down moves the leader, and connections open at that moment are reset while HAProxy re-routes to the new leader.

Operations

Checking Cluster State

Patroni’s own tool shows every member, its role, its state and its replication lag:

cpln workload exec RELEASE_NAME-postgres-ha --gvc GVC_NAME --replica RELEASE_NAME-postgres-ha-0 --container patroni-postgres -- patronictl -c /tmp/patroni_config.yml list

/tmp/patroni_config.yml is the configuration the startup script writes in each member; patronictl needs it to find etcd. The same tool shows the cluster-wide settings Patroni keeps in etcd:

cpln workload exec RELEASE_NAME-postgres-ha --gvc GVC_NAME --replica RELEASE_NAME-postgres-ha-0 --container patroni-postgres -- patronictl -c /tmp/patroni_config.yml show-config

For HAProxy’s view of the members — which one passes the primary health check — tunnel to the stats page and open http://127.0.0.1:8405/stats in a browser:

cpln port-forward RELEASE_NAME-postgres-ha-proxy 8405:8405 --gvc GVC_NAME

Cluster Metrics

From template version 2.7.0, Patroni’s Prometheus metrics are scraped into Control Plane’s built-in metrics stack automatically — there is no exporter to run and nothing to configure. They appear in the built-in Grafana alongside the standard workload metrics, labelled by workload, replica and container.

Metric	Tells you
`patroni_primary`	Which member holds the leader lock — `1` on exactly one member
`patroni_replica`	Which members are followers
`patroni_postgres_streaming`	Whether a follower is actually streaming from the leader
`patroni_xlog_location`	WAL position per member — the basis for replication lag
`patroni_postgres_timeline`	Timeline number; increments on every failover
`patroni_dcs_last_seen`	When this member last reached etcd
`patroni_failsafe_mode_is_active`	`1` while the leader is riding out an etcd outage instead of demoting
`patroni_cluster_unlocked`	`1` when no member holds the leader lock

Worth alerting on: patroni_primary summing to anything other than 1 across the cluster, patroni_cluster_unlocked staying 1, or patroni_postgres_streaming dropping to 0 on a follower. These are cluster-state metrics, not database performance — connection counts, cache hit ratio and slow queries would need a postgres_exporter, which this template does not include.

Backing Up

Backups are off by default. Enable them with backup.enabled: true, pick a backup.mode and a backup.provider, and complete that provider’s setup below first.

logical — RELEASE_NAME-postgres-ha-backup runs on backup.logical.schedule, connects through HAProxy with the username and password from your credentials secret, and uploads postgres-TIMESTAMP.sql.gz under backup.PROVIDER.prefix. HAProxy must be enabled.
wal-g — every member gets a wal-g-backup sidecar and PostgreSQL archives each WAL segment to the bucket as it completes (archive_timeout 60 s). The sidecar on whichever member is the leader takes a base backup every backup.walg.intervalSeconds, retrying every minute until one succeeds — including right after a failover. WAL archiving on its own cannot restore a cluster; it replays from a base backup. Confirm one exists before relying on this mode:
cpln workload exec RELEASE_NAME-postgres-ha --gvc GVC_NAME --container wal-g-backup -- wal-g backup-list
The sidecar does not prune old backups; manage retention in the bucket.

A zero-length logical backup object is a failed run, not a backup. If pg_dumpall cannot reach the cluster, the upload still writes a tiny empty gzip under a normal-looking timestamped name and the job exits non-zero. Check the object size before restoring from it — a real dump is kilobytes at minimum. Keep backup.resources.memory at the shipped 512Mi or above in wal-g mode: at 128Mi the sidecar is OOM-killed mid-backup (exitCode 137) and produces no base backup.

Backup Prerequisites

Complete the steps for your provider before enabling backups.

AWS S3

Create an S3 bucket. Set backup.aws.bucket to its name and backup.aws.region to its region.
If you do not have one yet, create a Control Plane cloud account for the AWS account holding the bucket. Set backup.aws.cloudAccountName to its name.
Create an IAM policy scoped to exactly that bucket, replacing YOUR_BUCKET_NAME:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:ListBucket",
                "s3:GetObjectVersion",
                "s3:DeleteObjectVersion",
                "s3:GetBucketLocation",
                "s3:AbortMultipartUpload",
                "s3:ListBucketMultipartUploads",
                "s3:ListMultipartUploadParts"
            ],
            "Resource": [
                "arn:aws:s3:::YOUR_BUCKET_NAME",
                "arn:aws:s3:::YOUR_BUCKET_NAME/*"
            ]
        }
    ]
}

Set backup.aws.policyName to that policy’s name. The template attaches it to the workload identity together with cpln-connector and nothing else — the bucket in your policy is the only storage the backup containers can reach. Upgrading from 2.5.0 or earlier narrows this further; see Upgrading From 2.5.0.

GCS

Create a GCS bucket. Set backup.gcp.bucket to its name.
Create a Control Plane cloud account for the GCP project holding the bucket, and set backup.gcp.cloudAccountName to its name.
Grant the cloud account’s service account the Storage Admin role. The template additionally binds the identity to roles/storage.objectAdmin on exactly the bucket in backup.gcp.bucket.

MinIO

No cloud account is needed — the backup containers authenticate with a second prerequisite secret. Both modes work: wal-g uses its native S3-compatible support with path-style addressing.

Create the bucket in MinIO. Set backup.minio.bucket to its name.
Set backup.minio.endpoint to the S3 API address including the port. For the minio template deployed in the same GVC that is http://WORKLOAD_NAME.GVC_NAME.cpln.local:9000.
Create a dictionary secret holding exactly the keys accessKey and secretKey, and set backup.minio.credentialsSecretName to its name. For the minio template these are its admin username and password:

cpln secret create-dictionary --name MINIO_CREDENTIALS_SECRET_NAME \
  --entry accessKey=MINIO_ACCESS_KEY \
  --entry secretKey=MINIO_SECRET_KEY

The policy this template creates grants the identity reveal on this secret as well as the database credentials secret. If the secret does not exist at install time the deployment wedges exactly as described in Prerequisites.

Restoring a Backup

Neither restore path has been rehearsed against this template. The logical commands are the standard way to replay a pg_dumpall script and match what the backup job writes; the wal-g outline follows the chart’s configuration and WAL-G’s documentation. Rehearse whichever you depend on against a scratch release before you need it.

Logical

Each backup is a gzip-compressed pg_dumpall script, so it is restored by connecting through HAProxy to the postgres maintenance database as the superuser from your credentials secret; the script recreates the roles and databases it contains, and HAProxy guarantees the writes land on the leader. Run the commands from a client that can reach both the bucket and the cluster — a workload inside the GVC, or your own machine with cpln port-forward RELEASE_NAME-postgres-ha-proxy 5432:5432 --gvc GVC_NAME open and --host=127.0.0.1 in place of the internal hostname.

AWS S3
GCS
MinIO

export PGPASSWORD="PASSWORD"

aws s3 cp "s3://BUCKET_NAME/PREFIX/postgres-TIMESTAMP.sql.gz" - \
  | gunzip \
  | psql \
      --host=RELEASE_NAME-postgres-ha-proxy.GVC_NAME.cpln.local \
      --port=5432 \
      --username=USERNAME \
      --dbname=postgres

unset PGPASSWORD

export PGPASSWORD="PASSWORD"

gsutil cp "gs://BUCKET_NAME/PREFIX/postgres-TIMESTAMP.sql.gz" - \
  | gunzip \
  | psql \
      --host=RELEASE_NAME-postgres-ha-proxy.GVC_NAME.cpln.local \
      --port=5432 \
      --username=USERNAME \
      --dbname=postgres

unset PGPASSWORD

export PGPASSWORD="PASSWORD"
export AWS_ACCESS_KEY_ID="MINIO_ACCESS_KEY"
export AWS_SECRET_ACCESS_KEY="MINIO_SECRET_KEY"
aws configure set default.s3.addressing_style path

aws s3 cp "s3://BUCKET_NAME/PREFIX/postgres-TIMESTAMP.sql.gz" - --endpoint-url "http://MINIO_ENDPOINT:9000" \
  | gunzip \
  | psql \
      --host=RELEASE_NAME-postgres-ha-proxy.GVC_NAME.cpln.local \
      --port=5432 \
      --username=USERNAME \
      --dbname=postgres

unset PGPASSWORD AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY

already exists errors for the superuser, the repl replication role and the database that the cluster created at first boot are expected; the rest of the script loads normally. Restore into an empty cluster (a fresh install, or one whose databases you have dropped) to avoid duplicate-object errors on existing tables. The replicas receive the restored data through normal streaming replication — nothing is needed on them.

WAL-G

WAL-G restores into an empty data directory, which this template has no automation for: a running member already has a data directory, and Patroni refuses to adopt a data directory whose system identifier does not match the cluster recorded in etcd. The procedure below is the shape of a manual restore. It is unverified on this template — treat it as an outline to rehearse, not a runbook.

List the base backups and pick one (LATEST is also accepted by backup-fetch):

cpln workload exec RELEASE_NAME-postgres-ha --gvc GVC_NAME --container wal-g-backup -- wal-g backup-list

Install a new release of this template into the GVC with backup.enabled: false, and stop it before Patroni initializes — or run a one-off workload of your own using the same image, with a fresh volume set mounted at /var/lib/postgresql/data and the same WALG_*/AWS_* environment the sidecar has (compare cpln workload get RELEASE_NAME-postgres-ha --gvc GVC_NAME -o yaml).
In that container, fetch the base backup into the empty PGDATA and configure recovery. The chart already sets restore_command: wal-g wal-fetch %f %p; for a point in time add recovery_target_time:
wal-g backup-fetch /var/lib/postgresql/data/pgdata BACKUP_NAME
Start PostgreSQL once on that data directory so it replays WAL to the target and finishes recovery, then point the new release’s members at the recovered volume and let Patroni initialize a new cluster from it.
Change backup.aws.prefix (or the GCS / MinIO prefix) before re-enabling wal-g backups on the restored cluster — it has a new system identifier and must not archive into the old cluster’s prefix.

If you cannot rehearse this, rely on logical backups for recovery and treat wal-g as an archive.

Rotating the Password

Patroni writes its configuration from the secret each time a member starts and applies the password to the database only when the cluster is first initialized. After that the stored passwords live in PostgreSQL, so a rotation touches three places, and the order matters. This procedure has not been rehearsed on this template — practise it on a scratch release first.

Update the password key in your credentials secret to the new value — for example with cpln apply -f of the full secret manifest — so any member that restarts from now on reads the new one. Nothing running changes yet: workloads resolve a secret only when a replica starts.
On the leader (through HAProxy), change both roles that share the password. The replication role repl was created with the same password as the superuser, and every member’s Patroni configuration will expect the new one after its next restart:
ALTER ROLE USERNAME WITH PASSWORD 'NEW-STRONG-PASSWORD'; ALTER ROLE repl WITH PASSWORD 'NEW-STRONG-PASSWORD';
Force a redeployment of the members, then of PgBouncer if it is enabled, so every process holds the new password. Expect a brief write outage while the members restart:
cpln workload force-redeployment RELEASE_NAME-postgres-ha --gvc GVC_NAME cpln workload force-redeployment RELEASE_NAME-pgbouncer --gvc GVC_NAME

The logical backup job reads the secret fresh on its next scheduled run and needs nothing else.

Upgrading From a Version Before 2.4.2

Template versions before 2.4.2 did not compact the bundled etcd cluster, so its backend grows with time alone and goes read-only once it reaches the 2 GiB quota — after which the PostgreSQL members restart-loop with exitCode: 0 and reason: Completed, which reads as healthy and is easily mistaken for a database fault. Upgrading turns compaction on and stops further growth, but it cannot shrink a backend that has already grown, and a cluster that has already raised a NOSPACE alarm needs operator recovery rather than an upgrade. Check first, with the read-only commands in If the Backend Quota Is Already Full, run against RELEASE_NAME-etcd.

Upgrading From 2.4.x

Template version 2.5.0 removed the postgres block. postgres.username, postgres.password and postgres.database are now the prerequisite dictionary secret named by config.credentialsSecretName, and MinIO backup credentials moved the same way. An upgrade that still carries any of the old keys is refused at render, before anything is applied, so your running cluster is left untouched:

postgres-highly-available: the `postgres` block was REMOVED in 2.5.0. `postgres.username`,
`postgres.password` and `postgres.database` are now a `dictionary` secret you create, named by
`config.credentialsSecretName`, holding the keys `username`, `password` and `database`. Delete the
`postgres:` block from your values. See Prerequisites in the README.

	2.4.x	2.5.0 and later
Database credentials	`postgres.username`, `postgres.password`, `postgres.database` values	`config.credentialsSecretName` → a dictionary secret you create
Where they were stored	`RELEASE_NAME-postgres-config`, created by the chart	Only in your secret; `RELEASE_NAME-postgres-config` now holds backup settings and is created only when backups are enabled
MinIO backup credentials	`backup.minio.accessKey`, `backup.minio.secretKey` values	`backup.minio.credentialsSecretName` → a dictionary secret you create

Read the credentials the cluster already uses

An existing cluster keeps the credentials it was initialized with. Recover them before upgrading, from your values file or from the secret the old version created:

cpln secret reveal RELEASE_NAME-postgres-config -o yaml

Create the prerequisite secret with those same values

Follow Prerequisites using the existing username, password and database. Different values here do not change the database — they leave every member unable to authenticate after its next restart.

Remove the old keys from your values

Delete the postgres: block and set config.credentialsSecretName. If you back up to MinIO, replace backup.minio.accessKey and backup.minio.secretKey with backup.minio.credentialsSecretName.

Upgrade, then rotate any default password

Versions 2.4.2 and earlier shipped username: username and password: password as defaults, published in the public template repository. If your cluster still runs them, treat them as compromised and follow Rotating the Password after the upgrade.

The chart no longer writes credentials into RELEASE_NAME-postgres-config. Anything outside this release that read that secret by name — another workload’s environment, a policy, a parent chart bundling this template — must be pointed at the secret you created instead. A reference to a key that no longer exists wedges the referring workload silently.

Upgrading From 2.5.0

Template version 2.5.1 removes aws::ReadOnlyAccess from the backup identity. That AWS managed policy granted read access to every bucket in your AWS account and contains no write actions, so it never carried the backup itself — but it was silently supplying any read action your own bucket-scoped policy happened to omit. Update your IAM policy to the full action list in AWS S3 before upgrading; if it already matches, no action is needed. The identity now carries cpln-connector and your bucket-scoped policy only.

Upgrading From a Version Before 2.6.0

Two things changed in 2.6.0 that an upgrade does not apply to a cluster that already exists:

Patroni’s consensus timeouts. The chart now initializes new clusters with ttl: 45, loop_wait: 10 and retry_timeout: 15. Earlier versions asked for ttl: 30 with retry_timeout: 30, a combination Patroni silently clamps rather than rejecting, so those clusters tolerated a shorter etcd outage than configured before demoting a healthy leader. Patroni reads these values only while the data directory is empty; afterwards they live in etcd, and an existing cluster keeps the ones it was created with. Move it with patronictl — all three together, because Patroni requires loop_wait + 2 × retry_timeout ≤ ttl and quietly rewrites the others when one is changed alone:
cpln workload exec RELEASE_NAME-postgres-ha --gvc GVC_NAME --replica RELEASE_NAME-postgres-ha-0 --container patroni-postgres -- patronictl -c /tmp/patroni_config.yml edit-config --force -s ttl=45 -s loop_wait=10 -s retry_timeout=15
Confirm with the show-config command in Checking Cluster State.
Enabling wal-g on an existing cluster. Before 2.6.0, turning on backup.mode: wal-g after the cluster was initialized did not start WAL archiving — only a fresh install archived. From 2.6.0 the archive settings apply to running members when they restart, so enabling wal-g on an upgraded cluster works; verify with wal-g backup-list as in Backing Up.

Also note that failsafe_mode is on for clusters created from 2.6.0: when the leader loses etcd but every member still answers its REST API, it keeps serving instead of demoting. A cluster created earlier gains it only through edit-config -s failsafe_mode=true.

Scaling and Availability

Leader loss fails over automatically. When the leader stops answering, Patroni promotes the most up-to-date replica and HAProxy’s health checks move port 5432 to it. Writes are refused in between and connections open at that moment are reset — build retry with backoff into your applications.
Planned restarts hand leadership off first. Before a leader is stopped, the chart asks Patroni to switch over to a replica, so a rolling restart costs a few seconds of refused writes rather than a full failover. A cpln helm upgrade that changes the member workload can restart members together; expect a brief write outage and plan upgrades as maintenance.
Losing etcd does not fail the cluster over. With failsafe_mode on, a leader that cannot reach etcd keeps serving as long as every other member still answers its REST API, and only a member Patroni recorded as eligible can win the next leader race. An etcd outage that also hides a replica still demotes the leader.
A member that stays down is re-cloned, not fatal. WAL retained for an absent member is capped at a quarter of volumeset.capacity; past that, its slot is dropped and it re-clones from the leader when it returns.
Keep replicas at 3 or more and etcd.replicas odd. Two members cannot fail over safely, and an even etcd count adds nothing to its fault tolerance. Choose etcd.replicas at install time.
Changing replicas on a running release adds or removes members; a removed member that held the leader role triggers a failover. Afterwards force a redeployment of RELEASE_NAME-postgres-ha-proxy so HAProxy picks up the new member list.
The first cpln helm upgrade after an install can re-apply resources that did not change and restart them. Plan it as a short outage.
multiZone: true on the members and on etcd spreads each across zones within the location, when the location supports it.

Troubleshooting

Workload never becomes ready and cpln logs returns nothing

Symptom: the install succeeds, RELEASE_NAME-postgres-ha stays not ready indefinitely, and cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-postgres-ha"}' --limit 50 --since 10m returns zero lines.Cause: config.credentialsSecretName (or, with MinIO backups, backup.minio.credentialsSecretName) names a secret that does not exist, so the container is never started. The message is only visible in status.versions[].message:

cpln workload get-deployments RELEASE_NAME-postgres-ha --gvc GVC_NAME -o yaml

The secret CREDENTIALS_SECRET_NAME no longer exists. Workload updates are paused until the
secret is added or the reference to the secret removed.

Fix: create the secret as in Prerequisites. The deployment recovers on its own within several minutes, or run cpln workload force-redeployment RELEASE_NAME-postgres-ha --gvc GVC_NAME.

Upgrade fails with 'the postgres block was REMOVED in 2.5.0'

Symptom: cpln helm upgrade stops before touching any resource:

Error: ... postgres-highly-available: the `postgres` block was REMOVED in 2.5.0. `postgres.username`,
`postgres.password` and `postgres.database` are now a `dictionary` secret you create, named by
`config.credentialsSecretName` ...

backup.minio.accessKey / backup.minio.secretKey produce the same kind of message under their own names.Cause: your values still carry keys that were removed in 2.5.0.Fix: follow Upgrading From 2.4.x. Your running cluster is unaffected until the upgrade succeeds.

HAProxy logs 'waiting for patroni endpoints to answer before starting haproxy'

Symptom: RELEASE_NAME-postgres-ha-proxy never becomes ready and cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-postgres-ha-proxy"}' --limit 50 --since 10m repeats pg-ha proxy: waiting for patroni endpoints to answer before starting haproxy — pgN (...) not responding yet.Cause: HAProxy will not start until every member’s Patroni REST API answers on port 8008. Either the members are still starting (normal for the first minutes of an install), or internal_access.type: workload-list does not include the proxy workload, so it cannot reach them.Fix: wait for RELEASE_NAME-postgres-ha to become ready. If it is ready and HAProxy still waits, add //gvc/GVC_NAME/workload/RELEASE_NAME-postgres-ha-proxy to internal_access.workloads as described in Internal Access. The container gives up and restarts after ten minutes, so it recovers on its own once the members are reachable.

Members restart-loop with exitCode 0 and reason Completed

Symptom: the PostgreSQL members restart repeatedly with a climbing restart count in cpln workload get-deployments RELEASE_NAME-postgres-ha --gvc GVC_NAME -o yaml, each exit reading exitCode: 0 and reason: Completed, while nothing in the database logs looks wrong.Cause: etcd has gone read-only. A Patroni member that cannot renew its lease exits cleanly, so the failure reads as healthy. The usual reason is an uncompacted etcd backend that has reached its 2 GiB quota and raised a NOSPACE alarm (template versions before 2.4.2).Fix: check etcd first, with two read-only commands:

cpln workload exec RELEASE_NAME-etcd --gvc GVC_NAME --container etcd -- etcdctl endpoint status --cluster -w table
cpln workload exec RELEASE_NAME-etcd --gvc GVC_NAME --container etcd -- etcdctl alarm list

A healthy cluster prints nothing for alarm list. NOSPACE confirms it; recovery is the operator procedure in If the Backend Quota Is Already Full, followed by an upgrade to 2.4.2 or later so it does not recur.

Clients get 'remaining connection slots are reserved' through PgBouncer

Symptom: under load, connections through RELEASE_NAME-pgbouncer fail with FATAL: remaining connection slots are reserved for roles with the SUPERUSER attribute (or ... for non-replication superuser connections) instead of waiting in the pool.Cause: pgbouncer.maxDbConnections is enforced per PgBouncer replica, so maxReplicas × maxDbConnections server connections can be opened against a cluster max_connections of 100 (97 usable).Fix: lower pgbouncer.maxDbConnections or pgbouncer.maxReplicas so their product stays under 97, and keep pgbouncer.defaultPoolSize within the same budget — see PgBouncer Connection Pooling.

wal-g backup-list shows no base backup, or the sidecar restarts with exitCode 137

Symptom: wal-g backup-list prints nothing hours after enabling backup.mode: wal-g, and cpln workload get-deployments RELEASE_NAME-postgres-ha --gvc GVC_NAME -o yaml shows the wal-g-backup container restarting with exitCode: 137.Cause: the sidecar is being OOM-killed mid-backup. Nothing is logged from inside the container when that happens; backup.resources.memory is below what wal-g backup-push needs.Fix: set backup.resources.memory to at least the shipped 512Mi and upgrade. The sidecar retries every minute, so a base backup appears shortly after it has enough memory.

Apply rejected with 'The ratio between cpu and minCpu must be less than 4:1'

Symptom: an install or upgrade fails and the member workload is not created or updated.Cause: resources.maxCpu is more than four times resources.minCpu, which the platform rejects on a stateful workload.Fix: raise resources.minCpu or lower resources.maxCpu so the ratio is at most 4:1.

Important Notes

Create the credentials secret before installing. A reference to a secret that does not exist wedges the members with no log output; see Prerequisites for the one command that shows the reason.
Connect through HAProxy (RELEASE_NAME-postgres-ha-proxy), or PgBouncer when enabled. A member reached directly may be a read-only replica.
Credentials are applied only when the cluster is first initialized, and the repl replication role shares the superuser password. Rotate in the order given in Rotating the Password.
Keep replicas at 3 or more and etcd.replicas odd, and choose etcd.replicas at install time.
Do not change etcd.tuning to disable compaction — an uncompacted etcd takes the whole cluster read-only months later.
With workload-list access, list the release’s own workloads as well as your applications, or HAProxy cannot reach the members.
Confirm a wal-g base backup exists with wal-g backup-list before relying on that mode; WAL alone cannot be restored.
Restore procedures are unverified on this template — rehearse the one you depend on against a scratch release.
cpln helm uninstall deletes both volume sets and the data with them. Your credentials secrets are yours and are left alone.
Upgrading from 2.4.x is a breaking change — the removed keys stop the upgrade rather than resetting a password. See Upgrading From 2.4.x.

External References

Patroni Documentation

Patroni configuration, REST API and patronictl reference

PostgreSQL Documentation

Official PostgreSQL 17 documentation

WAL-G for PostgreSQL

WAL-G commands, storage variables and restore guidance

PgBouncer Documentation

PgBouncer configuration reference

PostgreSQL HA Template

Source files, default values, and chart definition

​Overview

​What Gets Created

​Prerequisites

​Installation

UI

CLI

Terraform

Pulumi

​Configuration

​Cluster and Resources

​Credentials

​Storage

​Internal Access

​etcd Cluster

​etcd History Compaction

​HAProxy Leader Endpoint

​PgBouncer Connection Pooling

​Backup

​Connecting

​Operations

​Checking Cluster State

​Cluster Metrics

​Backing Up

​Backup Prerequisites

​AWS S3

​GCS

​MinIO

​Restoring a Backup

​Logical

​WAL-G

​Rotating the Password

​Upgrading From a Version Before 2.4.2

​Upgrading From 2.4.x

​Upgrading From 2.5.0

​Upgrading From a Version Before 2.6.0

​Scaling and Availability

​Troubleshooting

​Important Notes

​External References

Patroni Documentation

PostgreSQL Documentation

WAL-G for PostgreSQL

PgBouncer Documentation

PostgreSQL HA Template

Overview

What Gets Created

Prerequisites

Installation

Configuration

Cluster and Resources

Credentials

Storage

Internal Access

etcd Cluster

etcd History Compaction

HAProxy Leader Endpoint

PgBouncer Connection Pooling

Backup

Connecting

Operations

Checking Cluster State

Cluster Metrics

Backing Up

Backup Prerequisites

AWS S3

GCS

MinIO

Restoring a Backup

Logical

WAL-G

Rotating the Password

Upgrading From a Version Before 2.4.2

Upgrading From 2.4.x

Upgrading From 2.5.0

Upgrading From a Version Before 2.6.0

Scaling and Availability

Troubleshooting

Important Notes

External References