Skip to main content

Overview

Postgres Multi-Location deploys one PostgreSQL 17 Patroni cluster whose members span several Control Plane locations: a single primary that takes all traffic, asynchronous streaming replicas in the other locations, and automatic promotion of a replica in a surviving location when the primary’s location is lost. An HAProxy tier in every location routes connections to whichever member currently holds the leader lock, so applications connect to a stable name in their own region and never need to know where the primary is. Consensus is provided by a bundled etcd-multi-location cluster with one member per location — that quorum, not PostgreSQL, decides what the cluster survives.
For a cluster inside a single location, use PostgreSQL Highly Available instead.

How Many Locations You Need

The consensus store commits a write only when a majority of its members agree, and it runs exactly one member per location. With N locations you survive floor((N-1)/2) losses, so an even count buys nothing over the odd count below it. Two locations cannot form a symmetric quorum, which is why that topology is a warm standby rather than an automatic-failover cluster.

What Gets Created

  • GVC — A new GVC pinned to the configured locations. This template always creates it.
  • Stateful Patroni PostgreSQL Workload{release}-postgres-ml, running PostgreSQL 17 with Patroni. Each configured location runs replicas members, every member gets its own volume, and every member is individually addressable. PostgreSQL listens on 5432 and the Patroni REST API on 8008.
  • Stateful etcd Workload{release}-etcd, the bundled etcd-multi-location chart providing consensus, one member per location.
  • HAProxy Leader-Routing Workload (optional, enabled by default){release}-postgres-ml-proxy, one tier per location, each routing to the single current primary.
  • PgBouncer Workload (optional){release}-postgres-ml-pgbouncer, a connection pooler, one tier per location, pooling into that location’s HAProxy.
  • Cron Backup Workload (optional){release}-postgres-ml-backup, a scheduled pg_dumpall to object storage, running in exactly one location.
  • WAL-G Sidecar (optional) — A sidecar on every Patroni member; only the member holding the leader lock archives WAL and pushes base backups.
  • Volume Sets{release}-postgres-ml-vs for the PostgreSQL data directory (ext4, final snapshot with 7-day retention), plus the etcd cluster’s own volume set.
  • Secrets — Opaque secrets holding the Patroni startup script, the HAProxy startup script, and the WAL-G backup script. The database credentials secret is not created by this template — see Prerequisites.
  • Identity & Policy — An identity per workload group with reveal on exactly the secrets in play, plus a bucket-scoped cloud binding when backups are enabled.
This template creates its own GVC. You do not need to create one before installing.

Prerequisites

The GVC named in global.gvc.name must not already exist. Helm adopts a GVC that does, and helm uninstall then deletes it along with every unrelated workload in it. Always point this template at a fresh GVC name.
PostgreSQL credentials are supplied through a dictionary secret that you create before installing. They are never passed through Helm values.
1

Create the credentials secret

The secret must hold exactly three keys — username, password and database:
Use plain identifiers for username and database — they are used unquoted when the database is created. Secrets are org-level, so no GVC flag is involved.
2

Point the template at it

Set postgres.credentialsSecretName to the secret’s name.
3

Read the credentials back later

Create the secret before installing. Without it helm install still reports success, but the workload sits at zero replicas in every location with The secret ... no longer exists. Workload updates are paused until the secret is added or the reference to the secret removed. in cpln workload get-deployments. Creating the secret afterwards clears the wedge on its own within a few minutes — no reinstall or upgrade is needed.
Backups need additional setup before they can be enabled — see Backing Up. To install, follow the instructions for your preferred method:

UI

Browse, install, and manage templates visually

CLI

Manage templates from your terminal

Terraform

Declare templates in your Terraform configurations

Pulumi

Declare templates in your Pulumi programs

Configuration

The default values.yaml for this template:

GVC and Locations

The GVC name and the location list live under global.gvc so that Helm passes the same values to the bundled etcd cluster. Never configure the two location lists separately.
  • global.gvc.name — Name of the GVC this chart creates. It must not already exist.
  • global.gvc.locations[].name — A Control Plane location (e.g. aws-us-east-1). At least 2 are required; see How Many Locations You Need.
  • global.gvc.locations[].replicas — Number of Patroni members in that location. etcd always runs exactly one member per location regardless of this value. Members are named {workload}-{location}-{index}, so they stay unique when a location runs more than one.
Removing a location from the list is the supported way to shrink the cluster. Do not suspend a location instead — see Important Notes.

Credentials

postgres.credentialsSecretName names the dictionary secret described in Prerequisites. The template creates no credentials of its own, and PgBouncer reads the same secret.
The credentials are written into the data directory when the cluster first bootstraps. Rotating the secret afterwards does not change the database — change the password with ALTER ROLE first, then update the secret to match.

Preferred Primary Location

primaryLocation gives one location a higher failover priority than the others, so it wins the leader election when all members are eligible. Leaving it empty means no preference.
primaryLocation is baked into the Patroni startup script, so changing it on a live cluster rewrites that script and restarts every member. That costs the full upgrade write outage and the election that follows moves the leader. To move a primary without a restart, use patronictl switchover instead.
The preference biases elections; it does not fail back automatically after an outage. A cluster whose preferred location is restored keeps its current leader until the next election.

Resources and Storage

  • image — The Patroni + PostgreSQL 17 image.
  • resources.minCpu / resources.minMemory — Reserved CPU and memory per member.
  • resources.maxCpu / resources.maxMemory — Limits per member.
  • volumeset.capacity — Initial volume size in GiB (minimum 10). Every member gets its own volume.
  • volumeset.autoscaling.enabled — Expand the volume automatically as it fills. When enabled:
    • maxCapacity — Maximum volume size in GiB.
    • minFreePercentage — Trigger a scale-up when free space drops below this percentage.
    • scalingFactor — Multiply the current capacity by this factor when scaling up.

Internal Access

internalAccess applies to the PostgreSQL, HAProxy and PgBouncer tiers alike. There is no public access in this version — every endpoint is internal to Control Plane.
A firewall change is not instant. Closing access down was measured at under a minute, while re-opening it took over three minutes. Allow up to about four minutes and re-test before concluding the setting did not apply.

Leader-Routing Proxy

Only the member holding the leader lock accepts writes; the rest are read-only. HAProxy polls each member’s Patroni REST API and forwards connections to whichever one currently answers as the primary, so applications get a stable endpoint in their own location. proxy.minReplicas / proxy.maxReplicas are per location — the default of 2 means 2 replicas in each configured location.
The proxy is required for logical backups, which must dump from the current primary, and the chart refuses to render without it. Enabling PgBouncer enables the proxy implicitly, even if proxy.enabled is set to false.

PgBouncer Connection Pooling

PgBouncer multiplexes application connections into a smaller pool of real database connections. It pools into its own location’s HAProxy rather than into a member, so leader routing and failover stay transparent. When enabled it becomes the connection endpoint applications use.
  • pgbouncer.defaultPoolSize — Real PostgreSQL connections kept per PgBouncer replica (default 25).
  • pgbouncer.maxClientConn — Client connections accepted per PgBouncer replica (default 1000).
  • pgbouncer.maxDbConnections — Hard cap on total PostgreSQL connections, so scaling PgBouncer out cannot exhaust the primary.
  • pgbouncer.minReplicas / pgbouncer.maxReplicas — Autoscaling bounds, per location.

Connecting

Everything is internal to Control Plane. Substitute your release name and the GVC name from global.gvc.name. The proxy endpoint in every location reaches the same primary, wherever it is. Per-member names are useful for read-only queries against a nearby replica, or as a fallback when the proxy is disabled — but a client using them has to poll the members to discover which one is currently the primary.

Availability and Planned Outages

Measured on a three-location cluster (aws-us-east-1, aws-eu-central-1, aws-us-west-2) with one member per location and a one-second write probe running in each location.
Treat every helm upgrade as a planned write outage of roughly two minutes. Members do not restart one at a time: the field that would serialize the rollout is not retained by the platform, so all members go down together and no location can write during the window. This was measured on an upgrade that changed nothing at all, so it applies to any values change, including enabling a feature.
During an etcd outage Patroni’s failsafe mode keeps the primary serving writes — it held for about 60 seconds with zero failed writes. Once the replicas’ own health checks start failing, the failsafe loses its precondition and the primary demotes itself, which is where the ~19 s of unavailability comes from. The cluster re-promotes and resumes on its own; no operator action is needed.

Application Retry Logic

Applications must retry on transient database errors. Two are worth handling explicitly during a failover:
  • Connection-level failures — server closed the connection unexpectedly, terminating connection due to administrator command.
  • ERROR: cannot execute INSERT in a read-only transaction, seen briefly when the proxy forwards to a member that has just been demoted and its health check has not yet flipped.
Exponential backoff over a window of a few seconds covers a failover; an upgrade needs a window of a couple of minutes or a maintenance pause.

Data Loss on Failover

Replication is asynchronous. A promoted replica applies everything it has received, so a failover can lose the transactions that had not yet reached it — bounded by the replication lag at the instant of failure. Check lag with pg_stat_replication on the primary; in the service mesh every replication client reports the same client_addr, so use application_name (the member name) to tell members apart.

Operating the Cluster

Members are named {workload}-{location}-{index}, for example my-db-postgres-ml-aws-us-east-1-0. patronictl reads the config the startup script writes to /tmp/patroni_config.yml:
Consensus-level settings (ttl, loop_wait, retry_timeout, maximum_lag_on_failover, failsafe mode) are written once, when the cluster is first initialized, and are not values knobs. Change them on a live cluster with patronictl edit-config.

Recovering From a Lost Location

With 3 or more locations this section does not apply — losing one location is an automatic failover. With 2 locations, losing one loses consensus quorum permanently: the survivor holds current data but cannot be granted the leader lock, and consensus writes time out. To rebuild from the surviving member, set etcd.recovery.forceNewClusterInLocation to that member’s location and run a helm upgrade. Once writes are accepted again, return the value to "" and reprovision the failed location’s members — their volumes must be reset before they rejoin.

Backing Up

Backups are disabled by default. Set backup.enabled: true, choose a mode, and configure a storage provider. backup.location exists because a cron workload runs in every location of its GVC. Without it a three-location cluster would dump the same database three times a night into one bucket. Pick the location nearest the bucket — that is where the dump is read and uploaded from. It applies to logical mode only: the WAL-G sidecar runs on every member but only the one holding the leader lock pushes, so the archive follows the primary across a failover with no selector needed.
  • backup.logical.schedule — Cron expression for the dump (default: daily at 02:00 UTC). The dump runs through the proxy, so it always hits the current primary, and includes roles and all databases.
  • backup.walg.intervalSeconds — Seconds between base backups (default 21600, every 6 hours). WAL segments are archived continuously in between.
  • backup.resources.cpu / backup.resources.memory — Resources for whichever mode is enabled.
Switching backup.mode to or from wal-g restarts PostgreSQL, because it changes archive_mode. Enabling logical backups does not. After enabling wal-g, confirm SHOW archive_mode reports on in every location before relying on the archive — one location can take up to about ten minutes to pick up the new configuration.
The backup path has been exercised against AWS S3 only: dumps and WAL/base backups are written and verified in the bucket. A wal-g restore has been verified end to end (base backup plus WAL replay into an empty data directory, checksum-identical to source); the logical restore and the volume-set swap procedure below have not been exercised — the procedures below are documented, not tested. The gcp and minio providers are configured the same way but have not been exercised — validate them in a test install before relying on them. Likewise, rehearse your restore procedure in a scratch environment before you need it.

AWS S3

Before enabling backup with provider: aws, complete the following in your AWS account:
  1. Create an S3 bucket. Set backup.aws.bucket to its name and backup.aws.region to its region.
  2. If you do not have a Cloud Account set up, refer to the docs to Create a Cloud Account. Set backup.aws.cloudAccountName to its name.
  3. Create an IAM policy with the following JSON, replacing YOUR_BUCKET_NAME:
  1. Set backup.aws.policyName to the name of the policy created in step 3. This bucket-scoped policy is all the workload identity needs — no broad managed policy is required.
  2. Set backup.aws.prefix to the folder path where backups will be stored.

GCS

Before enabling backup with provider: gcp, complete the following in your GCP account:
  1. Create a GCS bucket. Set backup.gcp.bucket to its name.
  2. If you do not have a Cloud Account set up, refer to the docs to Create a Cloud Account. Set backup.gcp.cloudAccountName to its name.
  3. Add the Storage Admin role to the GCP service account associated with the Cloud Account. The template additionally binds the identity to roles/storage.objectAdmin on exactly the bucket named in backup.gcp.bucket.
  4. Set backup.gcp.prefix to the folder path where backups will be stored.

MinIO

No Cloud Account is needed — credentials are supplied as a secret.
  1. Create a bucket in MinIO. Set backup.minio.bucket to its name.
  2. Set backup.minio.endpoint to the MinIO S3 API address including the port. For the minio template deployed in the same GVC, use http://WORKLOAD_NAME:9000.
  3. Create a dictionary secret holding the MinIO credentials and set backup.minio.credentialsSecretName to its name. For the minio template these are its admin.username and admin.password:
  1. Set backup.minio.prefix to the folder path where backups will be stored.

Restoring a Backup

These procedures are documented from upstream, not exercised by us — no restore was performed during testing. Rehearse them in a scratch environment before you need them.

Logical

Stream the dump back through the proxy, which writes to the current primary. Run this from a client with access to the bucket:
For GCS, replace the aws s3 cp with gsutil cp "gs://BUCKET_NAME/PREFIX/BACKUP_FILE.sql.gz" -. For MinIO, add --endpoint-url "http://MINIO_ENDPOINT:9000" and run aws configure set default.s3.addressing_style path first.

WAL-G

A point-in-time restore needs an empty data directory, so restore into a new volume set:
  1. Run wal-g backup-list to identify the desired backup.
  2. Stop the Patroni workload.
  3. Create a new volume set and mount it at /var/lib/postgresql/data on a one-off restore workload.
  4. Run wal-g backup-fetch /var/lib/postgresql/data/pgdata BACKUP_NAME.
  5. Re-point the Patroni workload at the restored volume set and start it.
  6. Change the WAL-G prefix before re-enabling backups, or the new cluster’s WAL collides with the old system identifier.

Important Notes

  • Create the credentials secret before installing. postgres.credentialsSecretName names a secret this template does not create. Without it the deployment waits on a secret that does not exist; creating it afterwards clears the wedge within a few minutes.
  • The GVC in global.gvc.name must not already exist. Helm adopts an existing GVC and deletes it on uninstall, taking every unrelated workload with it.
  • Every helm upgrade interrupts writes in all locations for about two minutes. Plan changes as maintenance windows.
  • Replication is asynchronous, so a failover can lose the most recent transactions — bounded by the replication lag at the moment of failure.
  • primaryLocation moves a live primary. Changing it restarts every member, which costs the full upgrade outage and triggers an election. It does not fail back automatically after an outage.
  • Never suspend a location. Suspending and resuming one permanently withdraws its endpoints from the other locations’ service discovery while every status surface still reports healthy. Remove a location from global.gvc.locations instead.
  • Allow about two minutes after a cold install before believing a member is unreachable — cross-location service discovery takes that long to converge. Firewall changes can take up to about four minutes.
  • Consensus-level settings are not values knobs. They are written once, at first initialization; change them with patronictl edit-config.
  • Cost scales with write volume multiplied by the members outside the primary’s location. Each of them receives a full copy of the WAL stream, and cross-region traffic is billed. Read-mostly workloads are inexpensive to stretch; write-heavy ones are not.

External References

Patroni Documentation

Patroni clustering and automatic failover documentation

patronictl Reference

Command reference for inspecting and switching over a cluster

Patroni Dynamic Configuration

The consensus-level settings changed with patronictl edit-config

PostgreSQL 17 Documentation

Official PostgreSQL documentation

WAL-G Documentation

Continuous archiving and point-in-time recovery

etcd Documentation

Official etcd documentation

PgBouncer Documentation

PgBouncer configuration reference

Postgres Multi-Location Template

View the source files, default values, and chart definition