Overview
Postgres Multi-Location deploys one PostgreSQL 17 Patroni cluster whose members span several Control Plane locations: a single primary that takes all traffic, asynchronous streaming replicas in the other locations, and automatic promotion of a replica in a surviving location when the primary’s location is lost. An HAProxy tier in every location routes connections to whichever member currently holds the leader lock, so applications connect to a stable name in their own region and never need to know where the primary is. Consensus is provided by a bundledetcd-multi-location cluster with one member per location — that quorum, not PostgreSQL, decides what the cluster survives.
For a cluster inside a single location, use PostgreSQL Highly Available instead.
How Many Locations You Need
The consensus store commits a write only when a majority of its members agree, and it runs exactly one member per location.
With N locations you survive
floor((N-1)/2) losses, so an even count buys nothing over the odd count below it. Two locations cannot form a symmetric quorum, which is why that topology is a warm standby rather than an automatic-failover cluster.
What Gets Created
- GVC — A new GVC pinned to the configured locations. This template always creates it.
- Stateful Patroni PostgreSQL Workload —
{release}-postgres-ml, running PostgreSQL 17 with Patroni. Each configured location runsreplicasmembers, every member gets its own volume, and every member is individually addressable. PostgreSQL listens on5432and the Patroni REST API on8008. - Stateful etcd Workload —
{release}-etcd, the bundledetcd-multi-locationchart providing consensus, one member per location. - HAProxy Leader-Routing Workload (optional, enabled by default) —
{release}-postgres-ml-proxy, one tier per location, each routing to the single current primary. - PgBouncer Workload (optional) —
{release}-postgres-ml-pgbouncer, a connection pooler, one tier per location, pooling into that location’s HAProxy. - Cron Backup Workload (optional) —
{release}-postgres-ml-backup, a scheduledpg_dumpallto object storage, running in exactly one location. - WAL-G Sidecar (optional) — A sidecar on every Patroni member; only the member holding the leader lock archives WAL and pushes base backups.
- Volume Sets —
{release}-postgres-ml-vsfor the PostgreSQL data directory (ext4, final snapshot with 7-day retention), plus the etcd cluster’s own volume set. - Secrets — Opaque secrets holding the Patroni startup script, the HAProxy startup script, and the WAL-G backup script. The database credentials secret is not created by this template — see Prerequisites.
- Identity & Policy — An identity per workload group with
revealon exactly the secrets in play, plus a bucket-scoped cloud binding when backups are enabled.
This template creates its own GVC. You do not need to create one before installing.
Prerequisites
PostgreSQL credentials are supplied through a dictionary secret that you create before installing. They are never passed through Helm values.1
Create the credentials secret
The secret must hold exactly three keys — Use plain identifiers for
username, password and database:username and database — they are used unquoted when the database is created. Secrets are org-level, so no GVC flag is involved.2
Point the template at it
Set
postgres.credentialsSecretName to the secret’s name.3
Read the credentials back later
UI
Browse, install, and manage templates visually
CLI
Manage templates from your terminal
Terraform
Declare templates in your Terraform configurations
Pulumi
Declare templates in your Pulumi programs
Configuration
The defaultvalues.yaml for this template:
GVC and Locations
The GVC name and the location list live underglobal.gvc so that Helm passes the same values to the bundled etcd cluster. Never configure the two location lists separately.
global.gvc.name— Name of the GVC this chart creates. It must not already exist.global.gvc.locations[].name— A Control Plane location (e.g.aws-us-east-1). At least 2 are required; see How Many Locations You Need.global.gvc.locations[].replicas— Number of Patroni members in that location. etcd always runs exactly one member per location regardless of this value. Members are named{workload}-{location}-{index}, so they stay unique when a location runs more than one.
Credentials
postgres.credentialsSecretName names the dictionary secret described in Prerequisites. The template creates no credentials of its own, and PgBouncer reads the same secret.
The credentials are written into the data directory when the cluster first bootstraps. Rotating the secret afterwards does not change the database — change the password with
ALTER ROLE first, then update the secret to match.Preferred Primary Location
primaryLocation gives one location a higher failover priority than the others, so it wins the leader election when all members are eligible. Leaving it empty means no preference.
The preference biases elections; it does not fail back automatically after an outage. A cluster whose preferred location is restored keeps its current leader until the next election.
Resources and Storage
image— The Patroni + PostgreSQL 17 image.resources.minCpu/resources.minMemory— Reserved CPU and memory per member.resources.maxCpu/resources.maxMemory— Limits per member.volumeset.capacity— Initial volume size in GiB (minimum 10). Every member gets its own volume.volumeset.autoscaling.enabled— Expand the volume automatically as it fills. When enabled:maxCapacity— Maximum volume size in GiB.minFreePercentage— Trigger a scale-up when free space drops below this percentage.scalingFactor— Multiply the current capacity by this factor when scaling up.
Internal Access
internalAccess applies to the PostgreSQL, HAProxy and PgBouncer tiers alike. There is no public access in this version — every endpoint is internal to Control Plane.
A firewall change is not instant. Closing access down was measured at under a minute, while re-opening it took over three minutes. Allow up to about four minutes and re-test before concluding the setting did not apply.
Leader-Routing Proxy
Only the member holding the leader lock accepts writes; the rest are read-only. HAProxy polls each member’s Patroni REST API and forwards connections to whichever one currently answers as the primary, so applications get a stable endpoint in their own location.proxy.minReplicas / proxy.maxReplicas are per location — the default of 2 means 2 replicas in each configured location.
The proxy is required for
logical backups, which must dump from the current primary, and the chart refuses to render without it. Enabling PgBouncer enables the proxy implicitly, even if proxy.enabled is set to false.PgBouncer Connection Pooling
PgBouncer multiplexes application connections into a smaller pool of real database connections. It pools into its own location’s HAProxy rather than into a member, so leader routing and failover stay transparent. When enabled it becomes the connection endpoint applications use.pgbouncer.defaultPoolSize— Real PostgreSQL connections kept per PgBouncer replica (default25).pgbouncer.maxClientConn— Client connections accepted per PgBouncer replica (default1000).pgbouncer.maxDbConnections— Hard cap on total PostgreSQL connections, so scaling PgBouncer out cannot exhaust the primary.pgbouncer.minReplicas/pgbouncer.maxReplicas— Autoscaling bounds, per location.
Connecting
Everything is internal to Control Plane. Substitute your release name and the GVC name fromglobal.gvc.name.
The proxy endpoint in every location reaches the same primary, wherever it is. Per-member names are useful for read-only queries against a nearby replica, or as a fallback when the proxy is disabled — but a client using them has to poll the members to discover which one is currently the primary.
Availability and Planned Outages
Measured on a three-location cluster (aws-us-east-1, aws-eu-central-1, aws-us-west-2) with one member per location and a one-second write probe running in each location.
During an etcd outage Patroni’s failsafe mode keeps the primary serving writes — it held for about 60 seconds with zero failed writes. Once the replicas’ own health checks start failing, the failsafe loses its precondition and the primary demotes itself, which is where the ~19 s of unavailability comes from. The cluster re-promotes and resumes on its own; no operator action is needed.
Application Retry Logic
Applications must retry on transient database errors. Two are worth handling explicitly during a failover:- Connection-level failures —
server closed the connection unexpectedly,terminating connection due to administrator command. ERROR: cannot execute INSERT in a read-only transaction, seen briefly when the proxy forwards to a member that has just been demoted and its health check has not yet flipped.
Data Loss on Failover
Replication is asynchronous. A promoted replica applies everything it has received, so a failover can lose the transactions that had not yet reached it — bounded by the replication lag at the instant of failure. Check lag withpg_stat_replication on the primary; in the service mesh every replication client reports the same client_addr, so use application_name (the member name) to tell members apart.
Operating the Cluster
Members are named{workload}-{location}-{index}, for example my-db-postgres-ml-aws-us-east-1-0. patronictl reads the config the startup script writes to /tmp/patroni_config.yml:
ttl, loop_wait, retry_timeout, maximum_lag_on_failover, failsafe mode) are written once, when the cluster is first initialized, and are not values knobs. Change them on a live cluster with patronictl edit-config.
Recovering From a Lost Location
With 3 or more locations this section does not apply — losing one location is an automatic failover. With 2 locations, losing one loses consensus quorum permanently: the survivor holds current data but cannot be granted the leader lock, and consensus writes time out. To rebuild from the surviving member, setetcd.recovery.forceNewClusterInLocation to that member’s location and run a helm upgrade. Once writes are accepted again, return the value to "" and reprovision the failed location’s members — their volumes must be reset before they rejoin.
Backing Up
Backups are disabled by default. Setbackup.enabled: true, choose a mode, and configure a storage provider.
backup.location exists because a cron workload runs in every location of its GVC. Without it a three-location cluster would dump the same database three times a night into one bucket. Pick the location nearest the bucket — that is where the dump is read and uploaded from. It applies to logical mode only: the WAL-G sidecar runs on every member but only the one holding the leader lock pushes, so the archive follows the primary across a failover with no selector needed.
backup.logical.schedule— Cron expression for the dump (default: daily at 02:00 UTC). The dump runs through the proxy, so it always hits the current primary, and includes roles and all databases.backup.walg.intervalSeconds— Seconds between base backups (default21600, every 6 hours). WAL segments are archived continuously in between.backup.resources.cpu/backup.resources.memory— Resources for whichever mode is enabled.
Switching
backup.mode to or from wal-g restarts PostgreSQL, because it changes archive_mode. Enabling logical backups does not. After enabling wal-g, confirm SHOW archive_mode reports on in every location before relying on the archive — one location can take up to about ten minutes to pick up the new configuration.AWS S3
Before enabling backup withprovider: aws, complete the following in your AWS account:
- Create an S3 bucket. Set
backup.aws.bucketto its name andbackup.aws.regionto its region. - If you do not have a Cloud Account set up, refer to the docs to Create a Cloud Account. Set
backup.aws.cloudAccountNameto its name. - Create an IAM policy with the following JSON, replacing
YOUR_BUCKET_NAME:
- Set
backup.aws.policyNameto the name of the policy created in step 3. This bucket-scoped policy is all the workload identity needs — no broad managed policy is required. - Set
backup.aws.prefixto the folder path where backups will be stored.
GCS
Before enabling backup withprovider: gcp, complete the following in your GCP account:
- Create a GCS bucket. Set
backup.gcp.bucketto its name. - If you do not have a Cloud Account set up, refer to the docs to Create a Cloud Account. Set
backup.gcp.cloudAccountNameto its name. - Add the Storage Admin role to the GCP service account associated with the Cloud Account. The template additionally binds the identity to
roles/storage.objectAdminon exactly the bucket named inbackup.gcp.bucket. - Set
backup.gcp.prefixto the folder path where backups will be stored.
MinIO
No Cloud Account is needed — credentials are supplied as a secret.- Create a bucket in MinIO. Set
backup.minio.bucketto its name. - Set
backup.minio.endpointto the MinIO S3 API address including the port. For the minio template deployed in the same GVC, usehttp://WORKLOAD_NAME:9000. - Create a dictionary secret holding the MinIO credentials and set
backup.minio.credentialsSecretNameto its name. For theminiotemplate these are itsadmin.usernameandadmin.password:
- Set
backup.minio.prefixto the folder path where backups will be stored.
Restoring a Backup
These procedures are documented from upstream, not exercised by us — no restore was
performed during testing. Rehearse them in a scratch environment before you need them.
Logical
Stream the dump back through the proxy, which writes to the current primary. Run this from a client with access to the bucket:aws s3 cp with gsutil cp "gs://BUCKET_NAME/PREFIX/BACKUP_FILE.sql.gz" -. For MinIO, add --endpoint-url "http://MINIO_ENDPOINT:9000" and run aws configure set default.s3.addressing_style path first.
WAL-G
A point-in-time restore needs an empty data directory, so restore into a new volume set:- Run
wal-g backup-listto identify the desired backup. - Stop the Patroni workload.
- Create a new volume set and mount it at
/var/lib/postgresql/dataon a one-off restore workload. - Run
wal-g backup-fetch /var/lib/postgresql/data/pgdata BACKUP_NAME. - Re-point the Patroni workload at the restored volume set and start it.
- Change the WAL-G prefix before re-enabling backups, or the new cluster’s WAL collides with the old system identifier.
Important Notes
- Create the credentials secret before installing.
postgres.credentialsSecretNamenames a secret this template does not create. Without it the deployment waits on a secret that does not exist; creating it afterwards clears the wedge within a few minutes. - The GVC in
global.gvc.namemust not already exist. Helm adopts an existing GVC and deletes it on uninstall, taking every unrelated workload with it. - Every
helm upgradeinterrupts writes in all locations for about two minutes. Plan changes as maintenance windows. - Replication is asynchronous, so a failover can lose the most recent transactions — bounded by the replication lag at the moment of failure.
primaryLocationmoves a live primary. Changing it restarts every member, which costs the full upgrade outage and triggers an election. It does not fail back automatically after an outage.- Never suspend a location. Suspending and resuming one permanently withdraws its endpoints from the other locations’ service discovery while every status surface still reports healthy. Remove a location from
global.gvc.locationsinstead. - Allow about two minutes after a cold install before believing a member is unreachable — cross-location service discovery takes that long to converge. Firewall changes can take up to about four minutes.
- Consensus-level settings are not values knobs. They are written once, at first initialization; change them with
patronictl edit-config. - Cost scales with write volume multiplied by the members outside the primary’s location. Each of them receives a full copy of the WAL stream, and cross-region traffic is billed. Read-mostly workloads are inexpensive to stretch; write-heavy ones are not.
External References
Patroni Documentation
Patroni clustering and automatic failover documentation
patronictl Reference
Command reference for inspecting and switching over a cluster
Patroni Dynamic Configuration
The consensus-level settings changed with
patronictl edit-configPostgreSQL 17 Documentation
Official PostgreSQL documentation
WAL-G Documentation
Continuous archiving and point-in-time recovery
etcd Documentation
Official etcd documentation
PgBouncer Documentation
PgBouncer configuration reference
Postgres Multi-Location Template
View the source files, default values, and chart definition