Postgres Multi-Location - Control Plane

Version 2.0.0 no longer creates a GVC — it deploys into one you already have, and there is no in-place upgrade path from 1.x. A helm upgrade across that boundary deletes the GVC the old release created, and with it the volume sets holding your database. See Migrating From Version 1.

Overview

Postgres Multi-Location deploys one PostgreSQL 17 Patroni cluster whose members span several Control Plane locations: a single primary that takes all traffic, asynchronous streaming replicas in the other locations, and automatic promotion of a replica in a surviving location when the primary’s location is lost. An HAProxy tier in every location routes connections to whichever member currently holds the leader lock, so applications connect to a stable name in their own region and never need to know where the primary is. Consensus is provided by a bundled etcd-multi-location cluster with one member per location — that quorum, not PostgreSQL, decides what the cluster survives.

This template deploys into a GVC you already have and requires at least two locations. It does not create, provision or manage a GVC, and it cannot run in a single location: the consensus store is a stretched etcd cluster with exactly one member per location. For a single-location cluster, use PostgreSQL Highly Available instead.

How Many Locations You Need

The consensus store commits a write only when a majority of its members agree, and it runs exactly one member per location.

Locations	Majority	Location losses survived	What happens when one location is lost
2	2	0	The surviving replica holds current data but stays read-only. Promotion is manual.
3	2	1	Automatic failover — a replica in a surviving location is promoted and every proxy re-routes to it.
5	3	2	Survives losing two locations.

With N locations you survive floor((N-1)/2) losses, so an even count buys nothing over the odd count below it. Two locations cannot form a symmetric quorum, which is why that topology is a warm standby rather than an automatic-failover cluster.

What Gets Created

Stateful Patroni PostgreSQL Workload — {release}-postgres, running PostgreSQL 17 with Patroni. Each configured location runs replicas members, every member gets its own volume, and every member is individually addressable. PostgreSQL listens on 5432 and the Patroni REST API on 8008.
Stateful etcd Workload — {release}-etcd, the bundled etcd-multi-location chart providing consensus, one member per location.
HAProxy Leader-Routing Workload (optional, enabled by default) — {release}-postgres-proxy, one tier per location, each routing to the single current primary.
PgBouncer Workload (optional) — {release}-postgres-pgbouncer, a connection pooler, one tier per location, pooling into that location’s HAProxy.
Cron Backup Workload (optional) — {release}-postgres-backup, a scheduled pg_dumpall to object storage, running in exactly one location.
WAL-G Sidecar (optional) — A sidecar on every Patroni member; only the member holding the leader lock archives WAL and pushes base backups.
Volume Sets — {release}-postgres-vs for the PostgreSQL data directory (ext4, final snapshot with 7-day retention), plus the etcd cluster’s own volume set.
Secrets — Opaque secrets holding the Patroni startup script, the HAProxy startup script, and the WAL-G backup script. The database credentials secret is not created by this template — see Prerequisites.
Identities & Policies — An identity per workload group with reveal on exactly the secrets in play, view on exactly the one GVC this release installs into (for the boot-time location check described in Matching the Location List to the GVC), plus a bucket-scoped cloud binding when backups are enabled.

No GVC resource is created. Every resource above lands in the GVC you install into, and both startup scripts read that GVC’s name from the platform at runtime rather than from Helm, so a hostname can never drift from where the workload actually runs.

Prerequisites

An existing GVC with at least 2 locations, and global.locations set to match it. This template deploys into the GVC you install it into and creates none of its own. Read the GVC’s list with cpln gvc get GVC_NAME -o yaml and compare spec.staticPlacement.locationLinks before you install — see Matching the Location List to the GVC for what happens if the two disagree. PostgreSQL credentials are supplied through a dictionary secret that you create before installing. They are never passed through Helm values.

Create the credentials secret

The secret must hold exactly three keys — username, password and database:

cpln secret create-dictionary --name my-postgres-credentials \
  --entry username=postgres \
  --entry password="$(openssl rand -hex 24)" \
  --entry database=mydb

Use plain identifiers for username and database — they are used unquoted when the database is created. Secrets are org-level, so no GVC flag is involved.

Point the template at it

Set postgres.credentialsSecretName to the secret’s name.

Read the credentials back later

cpln secret reveal my-postgres-credentials

Create the secret before installing. Without it helm install still reports success, but the workload sits at zero replicas in every location with

The secret ... no longer exists. Workload updates are paused until the secret is added or the reference to the secret removed.

in cpln workload get-deployments. Creating the secret afterwards clears the wedge on its own within a few minutes — no reinstall or upgrade is needed.

Backups need additional setup before they can be enabled — see Backing Up. To install, follow the instructions for your preferred method:

UI

Browse, install, and manage templates visually

CLI

Manage templates from your terminal

Terraform

Declare templates in your Terraform configurations

Pulumi

Declare templates in your Pulumi programs

Configuration

The default values.yaml for this template:

# ─── Locations ────────────────────────────────────────────────────────────────
# This chart deploys into the GVC you install into — it does NOT create one.
# Every location listed here MUST already exist in that GVC. The platform does
# not validate that: a location the GVC lacks is stored verbatim and is simply
# inert, so the declared Patroni member and its etcd peer never start while every
# status surface reads healthy. The members read the GVC at boot and refuse to
# bootstrap in that state.
#
# Lives under `global` so the bundled etcd-multi-location subchart gets the same
# list automatically — the two lists can then never be edited apart.
#
# Minimum 2 locations. 3 gives automatic failover, 5 survives losing two;
# 2 gives a warm standby with MANUAL promotion. See the quorum table in the
# README. `replicas` is Patroni members per location; etcd always runs 1 per
# location and ignores this field.

global:
  locations:
    - name: aws-us-east-1
      replicas: 1
    - name: aws-eu-central-1
      replicas: 1
    - name: aws-us-west-2
      replicas: 1

# ─── PostgreSQL / Patroni ─────────────────────────────────────────────────────
image: controlplanecorporation/patroni-postgres:0.7

resources:
  minCpu: 500m
  minMemory: 1Gi
  maxCpu: 1
  maxMemory: 2Gi

postgres:
  # REQUIRED PREREQUISITE SECRET — CREATE IT BEFORE YOU INSTALL.
  # A `dictionary` secret holding exactly three keys: `username`, `password` and
  # `database`. If it does not exist at install time the deployment WEDGES
  # waiting on it and looks broken. See Prerequisites in the README for the
  # exact `cpln secret create-dictionary` command.
  credentialsSecretName: my-postgres-credentials

# Preferred location for the primary. It does three things:
#   1. On a FRESH install it decides where the primary starts — members in other
#      locations wait up to 90s for this one to initialise the cluster first.
#      If it is down or slow they bootstrap anyway (logged as a WARNING) and the
#      primary starts elsewhere; move it later with patronictl switchover.
#   2. It biases FAILOVER elections toward this location (failover_priority).
#   3. CHANGING IT ON A LIVE CLUSTER MOVES THE LEADER: the value is baked into
#      the startup script, so editing it restarts every member (~2 min of
#      interrupted writes) and the election that follows picks this location.
#      To move a primary WITHOUT a restart, use patronictl switchover.
# Empty = no preference, and the primary starts wherever a member gets there first.
primaryLocation: ""

volumeset:
  capacity: 10 # initial capacity in GiB (minimum is 10)
  autoscaling:
    enabled: false
    maxCapacity: 100 # GiB, when autoscaling is enabled
    minFreePercentage: 10 # free-space trigger
    scalingFactor: 1.2 # growth multiplier

internalAccess:
  type: same-gvc # options: same-gvc, same-org, workload-list
  # Only used when type is workload-list. This chart's OWN workloads (Patroni,
  # the proxy, PgBouncer, the backup cron) are added automatically — the list
  # also governs Patroni-to-Patroni replication and the proxy's health checks,
  # so a list naming only clients would cut the cluster off from itself. List
  # your clients here; do not list this release's workloads.
  workloads: []
    #- //gvc/GVC_NAME/workload/WORKLOAD_NAME

# ─── Leader-routing proxy ─────────────────────────────────────────────────────
# One HAProxy tier per location; every one routes to the single current primary.
proxy:
  enabled: true # automatically enabled when pgbouncer.enabled is true
  image: haproxy:2.9
  resources:
    cpu: 100m
    memory: 128Mi
  minReplicas: 2
  maxReplicas: 2

# ─── PgBouncer connection pooler ──────────────────────────────────────────────
# One tier per location, each pooling into its location's HAProxy.
pgbouncer:
  enabled: false
  image: edoburu/pgbouncer:v1.25.1-p0
  poolMode: transaction # options: session, transaction, statement
  defaultPoolSize: 25 # real Postgres connections PgBouncer keeps per replica
  maxClientConn: 1000 # client connections PgBouncer accepts per replica
  maxDbConnections: 100 # cap on Postgres connections PER PgBouncer pod — multiply by maxReplicas
  minReplicas: 2
  maxReplicas: 4
  resources:
    cpu: 200m
    memory: 128Mi

# ─── Backups ──────────────────────────────────────────────────────────────────
backup:
  enabled: false
  mode: logical # logical or wal-g
  # `logical` mode ONLY. A cron workload runs in EVERY location of its GVC, so
  # without this the job would fire once per location every night and write N
  # copies into one bucket. Pick the ONE location it runs in — nearest your
  # bucket. `wal-g` mode ignores it: that archives from whichever member is
  # currently the primary, wherever that is.
  location: aws-us-east-1
  resources: # applies to whichever mode is enabled
    cpu: 100m
    # 512Mi, not 128Mi: the GCP path OOMs at 128Mi with NO log output —
    # logical jobs merely report `failed`, and the wal-g sidecar loops on
    # OOMKilled while WAL archives with no base backup. AWS and MinIO are
    # fine at 128Mi; a default has to work for every provider.
    memory: 512Mi

  logical:
    image: ghcr.io/controlplane-com/backup-images/postgres-backup:17.1.0 # 17.1.0 = Postgres 17, 18.1.0 = Postgres 18
    schedule: "0 2 * * *" # cron schedule, default is daily at 02:00 UTC

  walg:
    intervalSeconds: 21600 # seconds between base backups, default is every 6 hours

  # storage settings are applied to whichever mode is enabled
  provider: aws # options: aws, gcp, minio

  aws:
    bucket: my-postgres-bucket
    region: us-east-1
    cloudAccountName: my-s3-cloud-account
    policyName: my-postgres-backup-policy # bucket-scoped IAM policy, see README
    prefix: postgres/backups # folder within the bucket

  gcp:
    bucket: my-postgres-bucket
    cloudAccountName: my-gcs-cloud-account
    prefix: postgres/backups # folder within the bucket

  minio: # a self-hosted MinIO workload, or any S3-compatible endpoint
    # MUST be reachable from EVERY location: wal-g's restore_command runs on
    # every member, and a member that cannot reach the archive never finishes
    # starting — its container is recycled every 160s indefinitely. A MinIO
    # workload in ONE location answers 503 from the others, so for wal-g use
    # a multi-location endpoint or an external S3-compatible service.
    endpoint: http://my-minio-workload:9000 # e.g. http://WORKLOAD.GVC.cpln.local:9000 in the same GVC
    bucket: my-postgres-bucket
    # REQUIRED PREREQUISITE SECRET when provider is `minio` — a `dictionary`
    # secret holding `accessKey` and `secretKey`. See Storage setup in the README.
    credentialsSecretName: my-postgres-minio-credentials
    prefix: postgres/backups # folder within the bucket

# ─── etcd (subchart: etcd-multi-location) ─────────────────────────────────────
etcd:
  image: controlplanecorporation/etcd:0.1
  resources:
    cpu: 500m
    memory: 512Mi
  tuning:
    heartbeatIntervalMs: 250
    electionTimeoutMs: 5000
    # History compaction. Without it etcd's backend grows with time alone —
    # Patroni renews its lease every ~10s and every renewal is a revision —
    # until it hits the quota and etcd goes READ-ONLY.
    autoCompactionMode: periodic # periodic (retention is a duration) or revision (a revision count)
    autoCompactionRetention: 1h # periodic needs an explicit unit (1h, 30m, 24h)
    quotaBackendBytes: 0 # backend size limit in bytes; 0 = etcd's own default of 2 GiB
  volumeset:
    capacity: 10
  internalAccess:
    type: same-gvc
    # etcd adds its OWN workload automatically, but not this chart's Patroni
    # workload. If you set type to workload-list here you MUST add
    # //gvc/YOUR_GVC/workload/RELEASE-postgres yourself, or Patroni loses its DCS.
    workloads: []
  recovery:
    # EMERGENCY ONLY — see "Recovering from a lost location" in the README.
    forceNewClusterInLocation: ""

Locations

The location list lives under global so that Helm passes the same list to the bundled etcd cluster. Never maintain the two lists separately.

global.locations[].name — A Control Plane location (e.g. aws-us-east-1) that must already be one of the locations of the GVC you install into. At least 2 are required; see How Many Locations You Need.
global.locations[].replicas — Number of Patroni members in that location. etcd always runs exactly one member per location regardless of this value. Members are named {workload}-{location}-{index}, so they stay unique when a location runs more than one.

Removing a location from the list is the supported way to shrink the cluster. Do not suspend a location instead — see Important Notes.

Matching the Location List to the GVC

The platform validates the pairing in neither direction, so the template closes both. Each mismatch fails differently:

Mismatch	What happens
The GVC has a location `global.locations` does not list	Nothing runs there. Every tier’s default `minScale`/`maxScale` are `0` and only the declared locations get a real count, so that location’s deployment reads `This workload location is deactivated because maxScale is set to 0`. Harmless, and it is how you deploy into a subset of a larger GVC.
`global.locations` lists a location the GVC does not have	The platform accepts and stores it, and it is simply inert — the declared Patroni member and its etcd peer never start. Left unchecked that quietly costs the cluster the fault tolerance it was sized for.

The members catch the second case themselves by reading their own GVC at boot, using the scoped view grant described in What Gets Created. The check is deliberately asymmetric:

On a fresh data directory it is a hard failure: etcd and Patroni both log a FATAL: line naming the missing location and refuse to bootstrap, and the containers exit with code 1.
On an already-initialized member it is a warning only, and the member keeps serving. A location removed from a GVC is indistinguishable from a location that is down, and surviving that is exactly what this template is for.
If the GVC read itself fails — a control-plane blip, or a missing policy — the check is skipped with a warning. It never stops a database on its own unavailability.

The same boot check also warns when primaryLocation or backup.location names a location the GVC lacks. Look for [patroni] and [etcd] lines in cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-postgres"}'.

Credentials

postgres.credentialsSecretName names the dictionary secret described in Prerequisites. The template creates no credentials of its own, and PgBouncer reads the same secret.

The credentials are written into the data directory when the cluster first bootstraps. Rotating the secret afterwards does not change the database — change the password with ALTER ROLE first, then update the secret to match.

Preferred Primary Location

primaryLocation names the location you want the primary in. Patroni treats “which member initializes an empty cluster” and “which member wins a failover” as separate questions, and the knob addresses both:

On a fresh install it biases where the primary starts. Since template version 1.0.2, a member outside the preferred location whose data directory is empty holds back before bootstrapping: it polls the preferred location’s members and proceeds as soon as one answers as the leader — or as a replica, which means the cluster is already initialized elsewhere and waiting cannot move it.
On a running cluster it biases failover elections toward that location through Patroni’s failover_priority tag, a tiebreaker between candidates that have replayed the same amount of WAL.

Leaving it empty means no preference: the primary starts wherever a member gets there first.

The head start is a bounded preference, not a guarantee. The wait is capped at 90 seconds, so if the preferred location is slow to start, down or misconfigured, another member bootstraps the cluster where it is and logs a WARNING naming the consequence. Move the primary afterwards with patronictl switchover --candidate. The wait cannot split the cluster: the consensus store is the arbiter, so a member that gives up early finds the cluster already initialized and clones as a replica instead. Before version 1.0.2 the knob rendered only the failover tag, so a fresh install raced and the primary could land in any location.

Check where the leader actually landed before judging a slow first install. Every write goes to the primary, so a primary outside primaryLocation puts a cross-region hop in front of your entire write path, and schema migrations are where that shows up first. On Grafana Multi-Location, which bundles this template, the same 713 migrations took about 5 seconds with the primary local and stretched to minutes when it bootstrapped in another region. In the pair of test installs measured that way, the cold install took 11 m 41 s with the primary remote against 4 m 19 s with it in place, and the Grafana instances waiting on the migrations restarted 15 times against 2 — every figure in that comparison comes from those two runs.

primaryLocation is baked into the Patroni startup script, so changing it on a live cluster rewrites that script and restarts every member. That costs the full upgrade write outage and the election that follows moves the leader. To move a primary without a restart, use patronictl switchover instead.

The preference biases elections; it does not fail back automatically after an outage. A cluster whose preferred location is restored keeps its current leader until the next election.

Resources and Storage

image — The Patroni + PostgreSQL 17 image.
resources.minCpu / resources.minMemory — Reserved CPU and memory per member.
resources.maxCpu / resources.maxMemory — Limits per member.
volumeset.capacity — Initial volume size in GiB (minimum 10). Every member gets its own volume.
volumeset.autoscaling.enabled — Expand the volume automatically as it fills. When enabled:
- maxCapacity — Maximum volume size in GiB.
- minFreePercentage — Trigger a scale-up when free space drops below this percentage.
- scalingFactor — Multiply the current capacity by this factor when scaling up.

Internal Access

internalAccess applies to the PostgreSQL, HAProxy and PgBouncer tiers alike. There is no public access in this version — every endpoint is internal to Control Plane.

Type	Description
`same-gvc`	Allow access from all workloads in the same GVC (default)
`same-org`	Allow access from all workloads in the same organization
`workload-list`	Allow access only from the workload links in `internalAccess.workloads`

With workload-list, list only your clients. The same list governs Patroni-to-Patroni streaming replication, the proxy’s health checks of every member, PgBouncer pooling into HAProxy and the nightly dump, so a list naming only clients would cut the cluster off from itself. This release’s own workloads — Patroni, the proxy, PgBouncer and the backup cron — are appended for you, and a client that is not listed is still refused.

The bundled etcd tier is a separate knob and is not covered by that. etcd.internalAccess has its own list, and the etcd chart appends only its own workload. If you set etcd.internalAccess.type: workload-list you must add //gvc/GVC_NAME/workload/RELEASE_NAME-postgres to etcd.internalAccess.workloads by hand, or Patroni loses its consensus store.

A firewall change is not instant. Blocking was measured at 47 seconds and unblocking at about 165 seconds on this template. Allow several minutes and re-test before concluding the setting did not apply.

Leader-Routing Proxy

Only the member holding the leader lock accepts writes; the rest are read-only. HAProxy polls each member’s Patroni REST API and forwards connections to whichever one currently answers as the primary, so applications get a stable endpoint in their own location. proxy.minReplicas / proxy.maxReplicas are per location — the default of 2 means 2 replicas in each configured location.

Endpoint	Description
`:8404/healthz`	Healthy when a primary backend is reachable, unhealthy otherwise
`:8405/stats`	Live HAProxy stats, including the health of each member

The proxy is required for logical backups, which must dump from the current primary, and the chart refuses to render without it. Enabling PgBouncer enables the proxy implicitly, even if proxy.enabled is set to false.

PgBouncer Connection Pooling

PgBouncer multiplexes application connections into a smaller pool of real database connections. It pools into its own location’s HAProxy rather than into a member, so leader routing and failover stay transparent. When enabled it becomes the connection endpoint applications use.

Mode	Description
`transaction`	Connection held for one transaction, then returned to the pool. Best for most web and API workloads. Not compatible with `SET` variables, temporary tables or advisory locks.
`session`	Connection held for the whole client session. Compatible with everything, less reuse — raise `defaultPoolSize` to your expected concurrency.
`statement`	Connection returned after every statement. Transactions are not supported.

pgbouncer.defaultPoolSize — Real PostgreSQL connections kept per PgBouncer replica (default 25).
pgbouncer.maxClientConn — Client connections accepted per PgBouncer replica (default 1000).
pgbouncer.maxDbConnections — Cap on PostgreSQL connections per PgBouncer replica, not across the tier. PgBouncer instances do not coordinate, so the real ceiling is maxReplicas × maxDbConnections; at the shipped values that is 4 × 100 = 400 against the cluster’s max_connections of 100, three of which are reserved for superusers. Size it so maxReplicas × maxDbConnections stays comfortably under 97, or clients get remaining connection slots are reserved under load rather than being queued. Treat defaultPoolSize the same way.
pgbouncer.minReplicas / pgbouncer.maxReplicas — Autoscaling bounds, per location.

etcd Consensus Store

The etcd block is passed straight through to the bundled etcd-multi-location chart — pinned to 2.0.0, the version that no longer creates a GVC — which is where its image, resources, storage, access and emergency recovery settings are documented. It reads its location list from the same global.locations, so the two can never be edited apart. etcd.tuning.autoCompactionMode, etcd.tuning.autoCompactionRetention and etcd.tuning.quotaBackendBytes control how much revision history etcd keeps and how large its backend may grow. Compaction is enabled in every version of the bundled chart and cannot be switched off, only retuned. The defaults — periodic, 1h and 0 (etcd’s own 2 GiB limit) — are the right settings for a Patroni consensus store and should be left alone: Patroni renews its leader lease every ten seconds or so, and every renewal is a revision, so an uncompacted backend grows with time alone until etcd goes read-only and takes the database’s failover with it. See Compaction and Backend Growth for the mechanism and the accepted value formats. etcd.tuning.heartbeatIntervalMs and etcd.tuning.electionTimeoutMs are the raft timers, tuned for cross-region round trips; raise both in proportion if your locations are more than about 250 ms apart.

Connecting

Everything is internal to Control Plane. Substitute your release name and the name of the GVC you installed the release into.

What	Where
PostgreSQL, pooled (when PgBouncer is enabled)	`RELEASE_NAME-postgres-pgbouncer.GVC_NAME.cpln.local:5432`
PostgreSQL, current primary (recommended otherwise)	`RELEASE_NAME-postgres-proxy.GVC_NAME.cpln.local:5432`
PostgreSQL, one specific member	`replica-INDEX.RELEASE_NAME-postgres.LOCATION.GVC_NAME.cpln.local:5432`
Patroni REST API	Port `8008` on the same per-member names (`/primary`, `/replica`, `/health`)
HAProxy health and stats	`:8404/healthz` and `:8405/stats` on the proxy workload
Credentials	The dictionary secret named by `postgres.credentialsSecretName`

The proxy endpoint in every location reaches the same primary, wherever it is. Per-member names are useful for read-only queries against a nearby replica, or as a fallback when the proxy is disabled — but a client using them has to poll the members to discover which one is currently the primary.

Migrating From Version 1

Every release before 2.0.0 created its own GVC. Do not helm upgrade a 1.x release onto 2.0.0. Once the chart stops declaring a GVC, Helm prunes the one the old release created — and deleting a GVC deletes every workload, volume set and identity inside it, including the volumes holding your database. Measured on a sibling template: everything was gone in about six seconds, while the command printed upgraded successfully.

The chart refuses to render if your values still carry the 1.x global.gvc key, so a values-carrying upgrade fails safely before any resource is touched. That guard cannot fire on an upgrade run with no values file at all, because it then sees only 2.0.0’s own defaults. The procedure below is the safety; the render guard is only a backstop.

Install 2.0.0 as a new release against an existing GVC, move the data across, then remove the old release.

Back up the 1.x cluster and verify the dump

Use backup.mode: logical, or run a pg_dumpall through the old release’s proxy endpoint. On 1.0.0 and 1.0.1 that endpoint is {release}-postgres-ml-proxy; from 1.0.2 onwards it is {release}-postgres-proxy. This dump is the only copy that crosses the boundary, so confirm it is a real dump and not a zero-length object before going further.

Rewrite your values

Delete global.gvc.name and rename global.gvc.locations to a top-level global.locations. Every location you list must already exist in the GVC you are installing into.

Install 2.0.0 as a new release into an existing GVC

Not the GVC the 1.x release created — that GVC is still owned by the old Helm release and goes away when you uninstall it.

Restore the dump and move your connection strings

Load the dump through the new release’s proxy — see Restoring a Backup — then repoint every application at {new-release}-postgres-proxy.GVC_NAME.cpln.local:5432.

Remove the old release

Uninstalling it removes the release and the GVC it created, taking the old volume sets with it. Do this only once the restore is verified.

Values that changed:

1.x	2.0.0
`global.gvc.name`	Removed — the GVC is wherever you install
`global.gvc.locations`	`global.locations`
Bundled `etcd-multi-location` 1.x	Pinned to 2.0.0, which also creates no GVC

Availability and Planned Outages

Measured on a three-location cluster (aws-us-east-1, aws-eu-central-1, aws-us-west-2) with one member per location and a one-second write probe running in each location.

Event	Measured impact
Clean install to all three members ready	169 s — leader elected, both replicas streaming with no lag. A later run of 2.0.0 with the proxy and pooler on measured 175 s for all four tiers
Graceful loss of the member holding the leader lock	Switchover in 3.19 s; writes unavailable 4.7–5.8 s per location; the old member rejoined as a streaming replica after 2 m 16 s
Abrupt loss of the primary (process hung, lease left to expire)	A replica in a surviving location was promoted within 43 s, bounded by the 45-second leader lease. Writes resumed through both the proxy and the pooler with no client reconfiguration, and the old primary rejoined as a streaming replica with no split brain
Any `helm upgrade`, including one that changes nothing	Writes unavailable in every location for about 117 s
Restart of the etcd tier	About 19 s of write unavailability; recovers automatically

Treat every helm upgrade as a planned write outage of roughly two minutes. Members do not restart one at a time: the field that would serialize the rollout is not retained by the platform, so all members go down together and no location can write during the window. This was measured on an upgrade that changed nothing at all, so it applies to any values change, including enabling a feature.

During an etcd outage Patroni’s failsafe mode keeps the primary serving writes — it held for about 60 seconds with zero failed writes. Once the replicas’ own health checks start failing, the failsafe loses its precondition and the primary demotes itself, which is where the ~19 s of unavailability comes from. The cluster re-promotes and resumes on its own; no operator action is needed.

Upgrading From 1.0.0 or 1.0.1

This section describes an upgrade within the 1.x line, where the chart still created its own GVC. Reaching 2.0.0 from any 1.x release is a migration rather than an upgrade — see Migrating From Version 1.

Version 1.0.2 renamed every PostgreSQL-side resource, there is no in-place upgrade path, and the upgrade DELETES the volume set holding your data. Back up before you go anywhere near it. The -ml infix is gone: the workloads, secrets, identity, policy and — critically — the volume set are now named {release}-postgres… instead of {release}-postgres-ml…. Running helm upgrade from 1.0.0 or 1.0.1 creates a new, empty volume set and then names the old one in its cleanup phase; it is gone seconds later, with no reachable final snapshot despite snapshots.createFinalSnapshot. There is no leftover copy to recover from.

Resource	1.0.0 / 1.0.1	1.0.2
Patroni workload	`{release}-postgres-ml`	`{release}-postgres`
Volume set (your data)	`{release}-postgres-ml-vs`	`{release}-postgres-vs`
HAProxy workload	`{release}-postgres-ml-proxy`	`{release}-postgres-proxy`
PgBouncer workload	`{release}-postgres-ml-pgbouncer`	`{release}-postgres-pgbouncer`
Backup workload	`{release}-postgres-ml-backup`	`{release}-postgres-backup`
Identity, policy, secrets	`{release}-postgres-ml-…`	`{release}-postgres-…`

The bundled etcd cluster’s resources were never renamed and are unaffected — which is the only reason a measured upgrade kept its data at all. Why the data survived a measured upgrade, and why that is not a plan. Helm creates the new workload before it deletes the old one, and because etcd’s volume set kept its name the cluster still had its DCS. All three new members therefore cloned the database from the still-running old primary (bootstrapped from leader …). That old primary kept serving for 86 seconds after Helm reported it deleted, and a 7.5 MB database cloned in 12 seconds. A database that cannot finish a base backup inside that window has nothing to fall back on, because the volume holding it has already been destroyed — that failure case is inferred, not observed, and it is not a race worth running. Two further consequences of the same upgrade, both measured:

No writable primary for about 2 minutes 22 seconds. The upgrade bounces the etcd tier too, so Patroni cannot immediately expire the stale leader key.
{release}-postgres-ml-proxy is deleted and never returns. Every application connection string has to move to {release}-postgres-proxy; nothing recovers on its own.

Migrate with a backup and restore rather than an upgrade:

Back up the existing cluster — first, and before anything else

Use backup.mode: logical, or run a pg_dumpall against the proxy endpoint of the old release. This dump is the only copy of your data that survives the rename, so verify it before you touch the release.

Uninstall the old release

cpln helm uninstall RELEASE_NAME --gvc GVC_NAME.

Install 1.0.2 and restore

Install the new version, then load the dump back through the proxy — see Restoring a Backup.

Move every client to the new proxy hostname

Update each application’s connection string from {release}-postgres-ml-proxy to {release}-postgres-proxy. The old hostname no longer exists.

Fresh installs of 1.0.2 are unaffected — this applies only to clusters created with an earlier version.

Application Retry Logic

Applications must retry on transient database errors. Two are worth handling explicitly during a failover:

Connection-level failures — server closed the connection unexpectedly, terminating connection due to administrator command.
ERROR: cannot execute INSERT in a read-only transaction, seen briefly when the proxy forwards to a member that has just been demoted and its health check has not yet flipped.

Exponential backoff over a window of a few seconds covers a failover; an upgrade needs a window of a couple of minutes or a maintenance pause.

Data Loss on Failover

Replication is asynchronous. A promoted replica applies everything it has received, so a failover can lose the transactions that had not yet reached it — bounded by the replication lag at the instant of failure. Check lag with pg_stat_replication on the primary; in the service mesh every replication client reports the same client_addr, so use application_name (the member name) to tell members apart.

Operating the Cluster

Members are named {workload}-{location}-{index}, for example my-db-postgres-aws-us-east-1-0. patronictl reads the config the startup script writes to /tmp/patroni_config.yml:

# Every member, its location, role and replication lag
cpln workload exec RELEASE_NAME-postgres --gvc GVC_NAME --container patroni-postgres \
  -- patronictl -c /tmp/patroni_config.yml list

# Move a live primary to another location — a planned, near-zero-downtime handover
cpln workload exec RELEASE_NAME-postgres --gvc GVC_NAME --container patroni-postgres \
  -- patronictl -c /tmp/patroni_config.yml switchover --candidate RELEASE_NAME-postgres-LOCATION-0 --force

Failover Timing

The cluster ships a fixed Patroni consensus configuration, and it is what bounds how long an abrupt loss of the primary takes to resolve:

Setting	Value	Meaning
`ttl`	`45` s	How long a dead primary’s leader lock survives before another member may claim it
`retry_timeout`	`15` s	How long the primary tolerates losing the consensus store before demoting itself
`loop_wait`	`10` s	How often the HA loop runs

Patroni enforces loop_wait + 2 × retry_timeout <= ttl and, when that is violated, silently substitutes values of its own rather than reporting an error — which is why the three always move together. Consensus-level settings (ttl, loop_wait, retry_timeout, maximum_lag_on_failover, failsafe mode) are written once, when the cluster is first initialized, and are not values knobs — a knob would look adjustable while only ever applying to a brand-new cluster. Read and change them on a live cluster with patronictl:

# What this cluster is actually running
cpln workload exec RELEASE_NAME-postgres --gvc GVC_NAME --container patroni-postgres \
  -- patronictl -c /tmp/patroni_config.yml show-config

# Change them — all three together
cpln workload exec RELEASE_NAME-postgres --gvc GVC_NAME --container patroni-postgres \
  -- patronictl -c /tmp/patroni_config.yml edit-config --force \
     -s ttl=45 -s loop_wait=10 -s retry_timeout=15

Recovering From a Lost Location

With 3 or more locations this section does not apply — losing one location is an automatic failover. With 2 locations, losing one loses consensus quorum permanently: the survivor holds current data but cannot be granted the leader lock, and consensus writes time out. To rebuild from the surviving member, set etcd.recovery.forceNewClusterInLocation to that member’s location and run a helm upgrade. Once writes are accepted again, return the value to "" and reprovision the failed location’s members — their volumes must be reset before they rejoin.

Backing Up

Backups are disabled by default. Set backup.enabled: true, choose a mode, and configure a storage provider.

Mode	Shape	Runs where	Good for
`logical`	Scheduled `pg_dumpall` cron workload	Exactly one location — `backup.location`	Portable SQL dumps, cross-version migration, smaller databases
`wal-g`	Sidecar on every member, active only on the primary	Follows the primary	Continuous WAL archiving and point-in-time recovery on larger databases

backup.location exists because a cron workload runs in every location of its GVC. Without it a three-location cluster would dump the same database three times a night into one bucket. Pick the location nearest the bucket — that is where the dump is read and uploaded from. It applies to logical mode only: the WAL-G sidecar runs on every member but only the one holding the leader lock pushes, so the archive follows the primary across a failover with no selector needed.

backup.logical.schedule — Cron expression for the dump (default: daily at 02:00 UTC). The dump runs through the proxy, so it always hits the current primary, and includes roles and all databases.
backup.walg.intervalSeconds — Seconds between base backups (default 21600, every 6 hours). WAL segments are archived continuously in between.
backup.resources.cpu / backup.resources.memory — Resources for whichever mode is enabled.

Switching backup.mode to or from wal-g restarts PostgreSQL, because it changes archive_mode. Enabling logical backups does not. After enabling wal-g, confirm SHOW archive_mode reports on in every location before relying on the archive — one location can take up to about ten minutes to pick up the new configuration.

Every provider has been exercised in both modes. AWS S3 and MinIO / S3-compatible passed in logical and wal-g mode alike, at the shipped settings. Google Cloud Storage works but is memory-sensitive: at backup.resources.memory: 128Mi it failed in both modes — a logical job simply reports failed with no error line anywhere, and the wal-g sidecar is OOM-killed in a loop, which takes its member out of service and triggers leader elections while WAL keeps accumulating with no base backup to restore it against. It passed at 256Mi (logical) and 512Mi (wal-g), and this template ships 512Mi, at or above both proven values — which is why the values comment warns against lowering it. Both restores have been verified end to end: a wal-g restore (base backup plus WAL replay into an empty data directory, checksum-identical to source) and a logical restore (a pg_dumpall artifact replayed into an initdb-fresh cluster with zero errors, roles, ownership and sequence positions preserved). The volume-set swap in the WAL-G procedure below has not been exercised — that step is documented, not tested. Rehearse your restore procedure in a scratch environment before you need it.

AWS S3

Before enabling backup with provider: aws, complete the following in your AWS account:

Create an S3 bucket. Set backup.aws.bucket to its name and backup.aws.region to its region.
If you do not have a Cloud Account set up, refer to the docs to Create a Cloud Account. Set backup.aws.cloudAccountName to its name.
Create an IAM policy with the following JSON, replacing YOUR_BUCKET_NAME:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:ListBucket",
                "s3:GetObjectVersion",
                "s3:DeleteObjectVersion"
            ],
            "Resource": [
                "arn:aws:s3:::YOUR_BUCKET_NAME",
                "arn:aws:s3:::YOUR_BUCKET_NAME/*"
            ]
        }
    ]
}

Set backup.aws.policyName to the name of the policy created in step 3. This bucket-scoped policy is all the workload identity needs — no broad managed policy is required.
Set backup.aws.prefix to the folder path where backups will be stored.

GCS

Before enabling backup with provider: gcp, complete the following in your GCP account:

Create a GCS bucket. Set backup.gcp.bucket to its name.
If you do not have a Cloud Account set up, refer to the docs to Create a Cloud Account. Set backup.gcp.cloudAccountName to its name.
Add the Storage Admin role to the GCP service account associated with the Cloud Account. The template additionally binds the identity to roles/storage.objectAdmin on exactly the bucket named in backup.gcp.bucket.
Set backup.gcp.prefix to the folder path where backups will be stored.

MinIO

In wal-g mode the endpoint must be reachable from every location. WAL-G’s restore_command runs on every member, and a member that cannot reach the archive never finishes starting — its container is recycled roughly every 160 seconds, indefinitely. A MinIO workload that runs in only one location answers 503 immediately from the others, which is exactly the shape a same-GVC MinIO invites: measured with two of three members looping for over 20 minutes. Nothing looks wrong from a client, because the leader stays healthy and writes keep succeeding while the cluster quietly loses all of its redundancy. For wal-g, point the endpoint at a MinIO deployment present in every location or at an external S3-compatible service. logical mode is unaffected — its cron runs only in backup.location — and AWS S3 and GCS are global endpoints, so they never hit this.

No Cloud Account is needed — credentials are supplied as a secret.

Create a bucket in MinIO. Set backup.minio.bucket to its name.
Set backup.minio.endpoint to the MinIO S3 API address including the port. For the minio template deployed in the same GVC, use http://WORKLOAD_NAME:9000.
Create a dictionary secret holding the MinIO credentials and set backup.minio.credentialsSecretName to its name. For the minio template these are its admin.username and admin.password:

cpln secret create-dictionary --name my-postgres-minio-credentials \
  --entry accessKey=MINIO_ACCESS_KEY \
  --entry secretKey=MINIO_SECRET_KEY

Set backup.minio.prefix to the folder path where backups will be stored.

Restoring a Backup

Both restore paths have been verified end to end, though not by running these exact commands: the pg_dumpall artifact was replayed with psql into an initdb-fresh cluster rather than through the proxy into a live one, and the WAL-G restore was fetched and replayed into an empty directory rather than onto a swapped volume set. The steps below are the operational form of those procedures — rehearse them in a scratch environment before you need them.

Logical

Stream the dump back through the proxy, which writes to the current primary. Run this from a client with access to the bucket:

export PGPASSWORD="PASSWORD"

aws s3 cp "s3://BUCKET_NAME/PREFIX/BACKUP_FILE.sql.gz" - \
  | gunzip \
  | psql \
      --host=RELEASE_NAME-postgres-proxy.GVC_NAME.cpln.local \
      --port=5432 \
      --username=USERNAME \
      --dbname=postgres

unset PGPASSWORD

For GCS, replace the aws s3 cp with gsutil cp "gs://BUCKET_NAME/PREFIX/BACKUP_FILE.sql.gz" -. For MinIO, add --endpoint-url "http://MINIO_ENDPOINT:9000" and run aws configure set default.s3.addressing_style path first.

WAL-G

The last two steps of this procedure cannot be carried out through the template as it ships. The Patroni workload’s volume set name is derived from the release name and hard-rendered into the workload, and values.yaml exposes only volumeset.capacity and volumeset.autoscaling.* — nothing that names or substitutes a volume set. Re-pointing the workload by hand would be reverted by the next helm upgrade. This is a known gap, reported rather than papered over with an untested replacement: mounting a restored volume is a manual, out-of-band operation today. The archive itself is genuinely restorable — a fetched base backup was verified as a valid PostgreSQL data directory whose system identifier matched the backup’s own metadata.

A point-in-time restore needs an empty data directory, so it restores into a new volume set:

Run wal-g backup-list to identify the desired backup. Verified working from the WAL-G sidecar.
Stop the Patroni workload.
Create a new volume set and mount it at /var/lib/postgresql/data on a one-off restore workload.
Run wal-g backup-fetch /var/lib/postgresql/data/pgdata BACKUP_NAME. Verified working — it produces a complete, valid data directory.
Re-point the Patroni workload at the restored volume set and start it. Not achievable through the chart — see the warning above.
Change the WAL-G prefix before re-enabling backups, or the new cluster’s WAL collides with the old system identifier.

Important Notes

There is no upgrade path from 1.x. Every 1.x release created its own GVC; 2.0.0 deploys into an existing one, and a helm upgrade across that boundary deletes the old GVC and every volume set in it. The chart refuses to render on the 1.x global.gvc key, but that guard cannot see an upgrade run with no values at all. Follow Migrating From Version 1: new release, restore, then uninstall the old one.
global.locations must match the GVC you install into. A GVC location this release does not list runs nothing, which is harmless; a listed location the GVC lacks makes a fresh cluster refuse to bootstrap, naming the location. See Matching the Location List to the GVC.
Back up before upgrading a 1.0.0 or 1.0.1 cluster to 1.0.2. Resource names dropped the -ml infix in 1.0.2, and the upgrade deletes the old volume set — there is no orphaned copy and no reachable final snapshot. Data survived a measured upgrade only because the new members re-cloned from the old primary during an 86-second window before it was torn down. Back up, uninstall, reinstall and restore instead, and move every connection string to {release}-postgres-proxy: see Upgrading From 1.0.0 or 1.0.1.
Create the credentials secret before installing. postgres.credentialsSecretName names a secret this template does not create. Without it the deployment waits on a secret that does not exist; creating it afterwards clears the wedge within a few minutes.
Every helm upgrade interrupts writes in all locations for about two minutes. Plan changes as maintenance windows.
Replication is asynchronous, so a failover can lose the most recent transactions — bounded by the replication lag at the moment of failure.
Set primaryLocation before the first install if you care where the primary is. Since version 1.0.2 it also biases where the primary bootstraps: members elsewhere wait up to 90 seconds for the preferred location, then bootstrap anyway and log a WARNING. Changing it later moves a live primary — that restarts every member, costs the full upgrade outage and triggers an election — and it does not fail back automatically after an outage.
Never suspend a location. Suspending and resuming one permanently withdraws its endpoints from the other locations’ service discovery while every status surface still reports healthy. To remove a location, take it out of global.locations and out of the GVC.
With internalAccess.type: workload-list, list only your clients — this release’s own tiers are added for you. The bundled etcd tier is separate: setting etcd.internalAccess.type: workload-list requires adding the Patroni workload link by hand, or Patroni loses its consensus store.
A restore from WAL-G cannot be completed through the chart today. The archive is restorable and backup-fetch produces a valid data directory, but nothing in values.yaml names the volume set the final step tells you to swap in. See WAL-G.
Allow about two minutes after a cold install before believing a member is unreachable — cross-location service discovery takes that long to converge. Firewall changes can take up to about four minutes.
Consensus-level settings are not values knobs. They are written once, at first initialization; change them with patronictl edit-config.
Cost scales with write volume multiplied by the members outside the primary’s location. Each of them receives a full copy of the WAL stream, and cross-region traffic is billed. Read-mostly workloads are inexpensive to stretch; write-heavy ones are not.

External References

Patroni Documentation

Patroni clustering and automatic failover documentation

patronictl Reference

Command reference for inspecting and switching over a cluster

Patroni Dynamic Configuration

The consensus-level settings changed with patronictl edit-config

PostgreSQL 17 Documentation

Official PostgreSQL documentation

WAL-G Documentation

Continuous archiving and point-in-time recovery

etcd Documentation

Official etcd documentation

PgBouncer Documentation

PgBouncer configuration reference

Postgres Multi-Location Template

View the source files, default values, and chart definition

​Overview

​How Many Locations You Need

​What Gets Created

​Prerequisites

UI

CLI

Terraform

Pulumi

​Configuration

​Locations

​Matching the Location List to the GVC

​Credentials

​Preferred Primary Location

​Resources and Storage

​Internal Access

​Leader-Routing Proxy

​PgBouncer Connection Pooling

​etcd Consensus Store

​Connecting

​Migrating From Version 1

​Availability and Planned Outages

​Upgrading From 1.0.0 or 1.0.1

​Application Retry Logic

​Data Loss on Failover

​Operating the Cluster

​Failover Timing

​Recovering From a Lost Location

​Backing Up

​AWS S3

​GCS

​MinIO

​Restoring a Backup

​Logical

​WAL-G

​Important Notes

​External References

Patroni Documentation

patronictl Reference

Patroni Dynamic Configuration

PostgreSQL 17 Documentation

WAL-G Documentation

etcd Documentation

PgBouncer Documentation

Postgres Multi-Location Template

Overview

How Many Locations You Need

What Gets Created

Prerequisites

Configuration

Locations

Matching the Location List to the GVC

Credentials

Preferred Primary Location

Resources and Storage

Internal Access

Leader-Routing Proxy

PgBouncer Connection Pooling

etcd Consensus Store

Connecting

Migrating From Version 1

Availability and Planned Outages

Upgrading From 1.0.0 or 1.0.1

Application Retry Logic

Data Loss on Failover

Operating the Cluster

Failover Timing

Recovering From a Lost Location

Backing Up

AWS S3

GCS

MinIO

Restoring a Backup

Logical

WAL-G

Important Notes

External References