> ## Documentation Index > Fetch the complete documentation index at: https://docs.controlplane.com/llms.txt > Use this file to discover all available pages before exploring further. # Postgres Multi-Location > One PostgreSQL 17 Patroni cluster stretched across locations: a single primary, streaming replicas elsewhere, automatic promotion when a location is lost, and a leader-routing proxy per location. Installs into an existing GVC with at least two locations; needs a credentials secret first. ## Overview Postgres Multi-Location deploys **one** PostgreSQL 17 cluster, managed by [Patroni](https://patroni.readthedocs.io/en/latest/), whose members span several Control Plane locations: a single primary that takes all writes, asynchronous streaming replicas in the other locations, and automatic promotion of a replica in a surviving location when the primary's location is lost. An HAProxy tier in every location routes connections to whichever member currently holds the leader lock, so applications connect to a stable name in their own region and never need to know where the primary is. An optional PgBouncer pooler per location and optional backups to object storage (scheduled `pg_dumpall`, or continuous WAL-G archiving) complete the stack. Consensus is provided by a bundled [etcd Multi-Location](/template-catalog/templates/etcd-multi-location) cluster with one member per location. That quorum — not PostgreSQL — decides what the cluster survives; see [Locations and Quorum](#locations-and-quorum). This template deploys into an existing [GVC](/reference/gvc) you already have; it does not create, provision or manage one. The GVC must have **at least two locations**, and every location you list in `global.locations` must already be one of them. A GVC location you do not list runs nothing — its deployment reads `This workload location is deactivated because maxScale is set to 0.` For a cluster in a single location use [PostgreSQL Highly Available](/template-catalog/templates/postgres-highly-available) instead; this template cannot run in one location because its consensus store is a stretched etcd cluster with exactly one member per location. ### What Gets Created | Resource | Name | Purpose | Optional | | - | - | - | - | | Workload | `RELEASE_NAME-postgres` | PostgreSQL 17 + Patroni members (stateful; `global.locations[].replicas` members per location). Container `patroni-postgres`: PostgreSQL on `5432`, Patroni REST API on `8008` | — | | Workload | `RELEASE_NAME-etcd` | The consensus store — the bundled etcd Multi-Location chart, exactly one member per location, client API on `2379` | — | | Workload | `RELEASE_NAME-postgres-proxy` | HAProxy leader-routing tier in every location (standard; container `haproxy`): `5432` to the current primary, `8404` health, `8405` stats | `proxy.enabled: true` (default; forced on when PgBouncer is enabled) | | Workload | `RELEASE_NAME-postgres-pgbouncer` | PgBouncer connection pooler in every location (standard; container `pgbouncer`), pooling into its own location's proxy on `5432` | `pgbouncer.enabled: true` | | Workload | `RELEASE_NAME-postgres-backup` | Scheduled `pg_dumpall` to S3, GCS or MinIO (cron; container `backup`), running in exactly one location | `backup.enabled: true` with `backup.mode: logical` | | Container | `wal-g-backup` | WAL-G sidecar added to every Patroni member; only the member holding the leader lock archives WAL and pushes base backups | `backup.enabled: true` with `backup.mode: wal-g` | | Volume Set | `RELEASE_NAME-postgres-vs` | `PGDATA` for every member — `ext4` on general-purpose SSD, final snapshot on deletion, snapshots retained 7 days | — | | Volume Set | `RELEASE_NAME-etcd-vs` | The etcd members' data directories | — | | Secret | `RELEASE_NAME-postgres-startup` | Patroni startup script (derives each member's name and peers at boot) | — | | Secret | `RELEASE_NAME-postgres-proxy-startup` | HAProxy startup script | with the proxy | | Secret | `RELEASE_NAME-postgres-wal-g` | WAL-G sidecar script | with `backup.mode: wal-g` | | Secret | `RELEASE_NAME-etcd-startup` | etcd startup script | — | | Identity | `RELEASE_NAME-postgres-identity` | Shared by the Patroni, proxy, pooler and backup workloads; carries the cloud account binding when backups use AWS or GCP | — | | Identity | `RELEASE_NAME-etcd-identity` | Bound to the etcd workload | — | | Policy | `RELEASE_NAME-postgres-policy` | Grants the identity `reveal` on your credentials secret, the startup scripts and (for MinIO) your MinIO credentials secret — nothing else | — | | Policy | `RELEASE_NAME-postgres-gvc-policy` | Grants the identity `view` on exactly the one GVC the release is installed into, so each member can confirm at boot that the GVC really has every declared location | — | | Policy | `RELEASE_NAME-etcd-policy`, `RELEASE_NAME-etcd-gvc-policy` | The same two grants for the etcd workload | — | The template creates **no credentials secret**. The database superuser, its password and the initial database come from a [dictionary secret](/guides/create-secret/dictionary) you create before installing (see [Prerequisites](#prerequisites)); the replication role `repl` uses the same password. Patroni members are named `RELEASE_NAME-postgres-LOCATION-INDEX` so that `patronictl list` maps directly onto your location list. ## Prerequisites Two things must exist before you install: a GVC with the right locations, and the credentials secret. Secrets are org-level, so no GVC flag is involved. This template deploys into an existing GVC and does not create one. Read the GVC's locations and compare them with the list you intend to install with: ```bash theme={null} cpln gvc get GVC_NAME -o yaml ``` `spec.staticPlacement.locationLinks` is the list. Every name you put in `global.locations` must appear there — a declared location the GVC lacks makes a fresh member refuse to start (see [Troubleshooting](#troubleshooting)). Three locations is the recommended production shape; see [Locations and Quorum](#locations-and-quorum). A [dictionary secret](/guides/create-secret/dictionary) holding exactly three keys — `username`, `password` and `database`. Patroni creates that superuser and that database when the cluster first bootstraps, and the replication role `repl` is given the same password. Use plain identifiers for `username` and `database`; they are used unquoted when the database is created. ```bash theme={null} cpln secret create-dictionary --name POSTGRES_CREDENTIALS_SECRET_NAME \ --entry username=postgres \ --entry password='YOUR-STRONG-PASSWORD' \ --entry database=mydb ``` Set `postgres.credentialsSecretName` to this name. Read it back later with `cpln secret reveal POSTGRES_CREDENTIALS_SECRET_NAME -o yaml` — a bare `cpln secret reveal` prints only a summary table, not the values. **Create the secret before installing, or the deployment wedges silently.** A `postgres.credentialsSecretName` that points at a secret which does not exist installs successfully and then never starts, in any location. The container never runs, so `cpln logs` returns zero lines. The one place the reason appears is `status.versions[].message`: ```bash theme={null} cpln workload get-deployments RELEASE_NAME-postgres --gvc GVC_NAME -o yaml ``` ```text theme={null} The secret POSTGRES_CREDENTIALS_SECRET_NAME no longer exists. Workload updates are paused until the secret is added or the reference to the secret removed. ``` Use `get-deployments` — plain `cpln workload get` has no `versions` key. Creating the missing secret repairs the deployment on its own within several minutes, or run `cpln workload force-redeployment RELEASE_NAME-postgres --gvc GVC_NAME` to skip the wait. Backups are optional and need a bucket plus, for AWS or GCP, a Control Plane [cloud account](/guides/create-cloud-account) — and for MinIO a second prerequisite secret. See [Backing Up](#backing-up). Nothing else is required for a default install. ## Installation The default `global.locations` list (`aws-us-east-1`, `aws-eu-central-1`, `aws-us-west-2`) is an example, not a recommendation — set it to your GVC's own locations in a values file, then install, pointing the chart at the secret you created: ```yaml theme={null} # postgres-locations.yaml — every name must already be a location of the GVC global: locations: - name: LOCATION_1 replicas: 1 - name: LOCATION_2 replicas: 1 - name: LOCATION_3 replicas: 1 ``` ```bash theme={null} cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/postgres-multi-location \ --version 2.0.0 --gvc GVC_NAME -f postgres-locations.yaml \ --set postgres.credentialsSecretName=POSTGRES_CREDENTIALS_SECRET_NAME ``` Use a values file for the location list rather than `--set`: a `--set global.locations[0].name=...` override replaces the whole list with a single entry, which the chart rejects. To install with another method, follow the instructions for your preferred tool: Browse, install, and manage templates visually Manage templates from your terminal }> Declare templates in your Terraform configurations Pulumi Icon Streamline Icon: https://streamlinehq.com } > Declare templates in your Pulumi programs ## Configuration Each block below is the shipped default for one area of `values.yaml`. Override only what you need with a values file or `--set`. ### Locations and Quorum ```yaml theme={null} global: locations: - name: aws-us-east-1 replicas: 1 - name: aws-eu-central-1 replicas: 1 - name: aws-us-west-2 replicas: 1 ``` * `global.locations[].name` — A Control Plane location that **must already be one of the locations of the GVC you install into**. At least two are required, each listed once. * `global.locations[].replicas` — Patroni members in that location (minimum `1`). etcd always runs exactly one member per location and ignores this field. The list lives under `global` so that Helm passes the same list to the bundled etcd chart; the two can never be edited apart. Removing a location from the list is the supported way to shrink the cluster — there is no per-location suspend, and a `replicas` of `0` is rejected at render time with a message telling you to remove the location instead. The consensus store commits a write only when a **majority** of its members agree, and it runs exactly one member per location. That arithmetic decides what the cluster survives: | Locations | Majority | Location losses survived | What happens when one location is lost | | - | - | - | - | | **2** | 2 | **0** | The surviving replica holds current data but stays **read-only**. Promotion is **manual** — see [Recovering From a Lost Location](#recovering-from-a-lost-location). | | **3** | 2 | **1** | **Automatic failover.** A replica in a surviving location is promoted and every proxy re-routes to it. The recommended shape. | | **4** | 3 | **1** | No better than 3, and costs more. | | **5** | 3 | **2** | Survives losing **two** locations. | With `N` locations you survive `floor((N-1)/2)` losses, so an even count buys nothing over the odd count below it. Two locations is permitted as a deliberate warm-standby topology, but it survives nothing automatically. The platform validates the pairing between `global.locations` and the GVC in neither direction, so the template closes both. A GVC location you do not list runs nothing (harmless — it is how you deploy into a subset of a larger GVC). A listed location the GVC does not have is accepted and stored, and the member that should run there never starts; the members catch this themselves by reading their own GVC at boot. On a **fresh** data directory that is a hard failure with a `[patroni] FATAL:` line naming the missing location; on an **already-initialized** member it is a warning only, because a location removed from a GVC looks exactly like a location that is down, and surviving that is what this template is for. See [Troubleshooting](#troubleshooting) for the exact messages. Changing `global.locations` on a running release restarts every Patroni member and every etcd member with a new roster. Plan it as a maintenance window and change the GVC's locations and this list together. ### Image and Resources ```yaml theme={null} image: controlplanecorporation/patroni-postgres:0.7 resources: minCpu: 500m minMemory: 1Gi maxCpu: 1 maxMemory: 2Gi ``` * `image` — PostgreSQL 17 with Patroni and WAL-G. The same image runs the WAL-G sidecar when `backup.mode: wal-g`. * `resources.minCpu` / `resources.minMemory` — Reserved CPU and memory per member. * `resources.maxCpu` / `resources.maxMemory` — Limits per member. On a stateful workload the ratio of `maxCpu` to `minCpu` may not exceed 4:1; a larger ratio is rejected when the workload is applied. The cluster ships fixed PostgreSQL settings (`max_connections: 100`, `shared_buffers: 128MB`, `wal_level: replica`, `hot_standby: on`, `max_slot_wal_keep_size: 10GB`) and is initialized with `UTF8` encoding and data checksums. Client authentication is password-based (`md5`) from any address; which workloads can reach the cluster at all is decided by [Internal Access](#internal-access). ### Credentials ```yaml theme={null} postgres: # REQUIRED PREREQUISITE SECRET — CREATE IT BEFORE YOU INSTALL. # A `dictionary` secret holding exactly three keys: `username`, `password` and # `database`. If it does not exist at install time the deployment WEDGES # waiting on it and looks broken. See Prerequisites in the README for the # exact `cpln secret create-dictionary` command. credentialsSecretName: my-postgres-credentials ``` `postgres.credentialsSecretName` names the dictionary secret described in [Prerequisites](#prerequisites). The Patroni members, the WAL-G sidecar, PgBouncer and the backup job all read the same secret. The three keys are applied only when the cluster first bootstraps; changing the secret afterwards does not change the stored password — see [Rotating the Password](#rotating-the-password). ### Preferred Primary Location ```yaml theme={null} primaryLocation: "" ``` `primaryLocation` names the location you want the primary in. Leave it empty for no preference, in which case the primary starts wherever a member gets there first. When set, it must be one of the names in `global.locations` and does three things: * **On a fresh install it decides where the primary starts.** Members outside the preferred location wait up to 90 seconds for that location to initialize the cluster before bootstrapping themselves. If the preferred location is down, slow or misconfigured they bootstrap anyway and log `WARNING: no leader in LOCATION after 90s — bootstrapping here instead`; the cluster is then correct but the primary is elsewhere, and you move it with `patronictl switchover` (see [Managing the Cluster With patronictl](#managing-the-cluster-with-patronictl)). The wait cannot split the cluster: the consensus store is the arbiter, so a member that gives up early finds the cluster already initialized and joins it as a replica. * **It biases failover elections** toward that location through Patroni's `failover_priority` tag, a tiebreaker between candidates that have replayed the same amount of WAL. It does not fail back automatically after an outage: a restored preferred location keeps the current leader until the next election. * **Changing it on a live cluster moves the leader.** The value is baked into the startup script, so editing it restarts every member, and the election that follows picks the preferred location. That is a full restart with a write outage — to move a primary without one, use `patronictl switchover` instead. ### Storage ```yaml theme={null} volumeset: capacity: 10 # initial capacity in GiB (minimum is 10) autoscaling: enabled: false maxCapacity: 100 # GiB, when autoscaling is enabled minFreePercentage: 10 # free-space trigger scalingFactor: 1.2 # growth multiplier ``` * `volumeset.capacity` — Initial size in GiB of the volume every member gets for `PGDATA` (minimum 10). The volume set keeps snapshots for 7 days and takes a final snapshot when it is deleted. * `volumeset.autoscaling.*` — Grow the volume automatically when free space drops below `minFreePercentage`, multiplying the capacity by `scalingFactor` up to `maxCapacity`. The growth event itself has not been exercised on this template. ### Internal Access ```yaml theme={null} internalAccess: type: same-gvc # options: same-gvc, same-org, workload-list # Only used when type is workload-list. This chart's OWN workloads (Patroni, # the proxy, PgBouncer, the backup cron) are added automatically — the list # also governs Patroni-to-Patroni replication and the proxy's health checks, # so a list naming only clients would cut the cluster off from itself. List # your clients here; do not list this release's workloads. workloads: [] #- //gvc/GVC_NAME/workload/WORKLOAD_NAME ``` `internalAccess` applies to the PostgreSQL, HAProxy and PgBouncer tiers alike. There is no public access — every endpoint is internal to Control Plane. | Type | Who can reach port `5432` | | - | - | | `same-gvc` | Every workload in the same GVC (default). Traffic between locations inside one GVC is same-GVC traffic, so a stretched cluster needs no extra rule. | | `same-org` | Every workload in the same org | | `workload-list` | Only the workload links in `internalAccess.workloads`, plus this release's own workloads, which the template appends for you | **The bundled etcd tier is a separate knob and is not covered by that.** `etcd.internalAccess` has its own list, and the etcd chart appends only its *own* workload. If you set `etcd.internalAccess.type: workload-list` you must add `//gvc/GVC_NAME/workload/RELEASE_NAME-postgres` to `etcd.internalAccess.workloads` by hand, or Patroni loses its consensus store. A change to `internalAccess` can take several minutes to take effect in every location — re-test before concluding it did not apply. ### Leader-Routing Proxy ```yaml theme={null} proxy: enabled: true # automatically enabled when pgbouncer.enabled is true image: haproxy:2.9 resources: cpu: 100m memory: 128Mi minReplicas: 2 maxReplicas: 2 ``` Only the member holding the leader lock accepts writes; the rest are read-only. HAProxy checks every member's Patroni REST API (`GET /primary` on port `8008`) and forwards port `5432` to whichever one currently answers as the primary, so applications get a stable endpoint in their own location that follows the leader across a failover. * `proxy.minReplicas` / `proxy.maxReplicas` — Autoscaling bounds **per location**; the default `2` means two proxy replicas in each configured location. * Port `8404` serves `/healthz` (healthy while a primary backend is reachable) and port `8405` serves `/stats`, the live HAProxy status of every member. The proxy is required for `logical` backups, which must dump from the current primary; the chart refuses to render `backup.mode: logical` with the proxy disabled. Enabling PgBouncer enables the proxy implicitly, even if `proxy.enabled` is `false`. ### PgBouncer ```yaml theme={null} pgbouncer: enabled: false image: edoburu/pgbouncer:v1.25.1-p0 poolMode: transaction # options: session, transaction, statement defaultPoolSize: 25 # real Postgres connections PgBouncer keeps per replica maxClientConn: 1000 # client connections PgBouncer accepts per replica maxDbConnections: 100 # cap on Postgres connections PER PgBouncer pod — multiply by maxReplicas minReplicas: 2 maxReplicas: 4 resources: cpu: 200m memory: 128Mi ``` PgBouncer multiplexes application connections into a smaller pool of real database connections. It pools into its own location's proxy rather than into a member, so leader routing and failover stay transparent to the pooler, and it reads the same credentials secret as the server. When enabled it becomes the endpoint applications use. | `poolMode` | Behaviour | | - | - | | `transaction` | The server connection is held for one transaction. Best for most web and API workloads; not compatible with `SET` variables, temporary tables or advisory locks. | | `session` | Held for the whole client session. Compatible with everything, less reuse — raise `defaultPoolSize` to your expected concurrency. | | `statement` | Returned after every statement. Transactions are not supported. | * `pgbouncer.minReplicas` / `pgbouncer.maxReplicas` — Autoscaling bounds **per location**. * `pgbouncer.maxDbConnections` — Enforced **per PgBouncer replica**, not across the tier. PgBouncer instances do not coordinate, so the real ceiling on server connections is `maxReplicas × maxDbConnections` — at the shipped values 4 × 100 = 400 against the cluster's `max_connections` of 100, three of which are reserved for superusers. Size it so `maxReplicas × maxDbConnections` stays comfortably under 97, and treat `defaultPoolSize` the same way; otherwise clients get `remaining connection slots are reserved` under load instead of being queued. ### Backups ```yaml theme={null} backup: enabled: false mode: logical # logical or wal-g # `logical` mode ONLY. A cron workload runs in EVERY location of its GVC, so # without this the job would fire once per location every night and write N # copies into one bucket. Pick the ONE location it runs in — nearest your # bucket. `wal-g` mode ignores it: that archives from whichever member is # currently the primary, wherever that is. location: aws-us-east-1 resources: # applies to whichever mode is enabled cpu: 100m memory: 512Mi logical: image: ghcr.io/controlplane-com/backup-images/postgres-backup:17.1.0 # 17.1.0 = Postgres 17, 18.1.0 = Postgres 18 schedule: "0 2 * * *" # cron schedule, default is daily at 02:00 UTC walg: intervalSeconds: 21600 # seconds between base backups, default is every 6 hours # storage settings are applied to whichever mode is enabled provider: aws # options: aws, gcp, minio aws: bucket: my-postgres-bucket region: us-east-1 cloudAccountName: my-s3-cloud-account policyName: my-postgres-backup-policy # bucket-scoped IAM policy, see README prefix: postgres/backups # folder within the bucket gcp: bucket: my-postgres-bucket cloudAccountName: my-gcs-cloud-account prefix: postgres/backups # folder within the bucket minio: # a self-hosted MinIO workload, or any S3-compatible endpoint endpoint: http://my-minio-workload:9000 # e.g. http://WORKLOAD.GVC.cpln.local:9000 in the same GVC bucket: my-postgres-bucket # REQUIRED PREREQUISITE SECRET when provider is `minio` — a `dictionary` # secret holding `accessKey` and `secretKey`. See Storage setup in the README. credentialsSecretName: my-postgres-minio-credentials prefix: postgres/backups # folder within the bucket ``` | `mode` | Shape | Runs where | Good for | | - | - | - | - | | `logical` | Scheduled `pg_dumpall` cron workload, dumping through the proxy | Exactly one location — `backup.location`, which must be one of `global.locations` | Portable SQL dumps, cross-version migration, smaller databases | | `wal-g` | Sidecar on every member, active only on the primary | Follows the primary | Continuous WAL archiving plus periodic base backups on larger databases | * `backup.location` — `logical` mode only. A cron workload runs in every location of its GVC, so without this selector a three-location cluster would dump the same database three times a night into one bucket. The WAL-G sidecar needs no selector: only the member holding the leader lock pushes, so the archive follows the primary across a failover. * `backup.resources.memory` — Do not lower it below the shipped `512Mi`. The Google Cloud Storage path fails at `128Mi` with no error line anywhere — a `logical` job simply reports `failed`, and the `wal-g` sidecar is killed in a loop while WAL accumulates with no base backup to restore against. * `backup.provider` and the `aws` / `gcp` / `minio` blocks apply to whichever mode is enabled. Each provider's bucket, cloud account, policy and secret steps are under [Backing Up](#backing-up). Switching `backup.mode` to or from `wal-g` restarts PostgreSQL in every location, because it changes `archive_mode`. Enabling `logical` backups does not restart anything. ### etcd Consensus Store ```yaml theme={null} etcd: image: controlplanecorporation/etcd:0.1 resources: cpu: 500m memory: 512Mi tuning: heartbeatIntervalMs: 250 electionTimeoutMs: 5000 # History compaction. Without it etcd's backend grows with time alone — # Patroni renews its lease every ~10s and every renewal is a revision — # until it hits the quota and etcd goes READ-ONLY. autoCompactionMode: periodic # periodic (retention is a duration) or revision (a revision count) autoCompactionRetention: 1h # periodic needs an explicit unit (1h, 30m, 24h) quotaBackendBytes: 0 # backend size limit in bytes; 0 = etcd's own default of 2 GiB volumeset: capacity: 10 internalAccess: type: same-gvc # etcd adds its OWN workload automatically, but not this chart's Patroni # workload. If you set type to workload-list here you MUST add # //gvc/YOUR_GVC/workload/RELEASE-postgres yourself, or Patroni loses its DCS. workloads: [] recovery: # EMERGENCY ONLY — see "Recovering from a lost location" in the README. forceNewClusterInLocation: "" ``` The `etcd` block is passed straight through to the bundled [etcd Multi-Location](/template-catalog/templates/etcd-multi-location) chart, pinned to **2.0.0**, which is where its image, resources, Raft timers, storage and access settings are documented in full. It reads its location list from the same `global.locations`. * `etcd.tuning.autoCompactionMode` / `autoCompactionRetention` / `quotaBackendBytes` — Leave at the defaults. Patroni renews its leader lease every few seconds and every renewal is an etcd revision, so an uncompacted backend grows with time alone until etcd goes read-only and takes the database's failover with it. Compaction cannot be switched off, only retuned; see [Compaction and Backend Growth](/template-catalog/templates/etcd-multi-location#compaction-and-backend-growth) for the mechanism and the accepted value formats. * `etcd.tuning.heartbeatIntervalMs` / `electionTimeoutMs` — The Raft timers, tuned for cross-region round trips; raise both in proportion if your locations are more than about 250 ms apart. * `etcd.recovery.forceNewClusterInLocation` — Leave empty in normal operation. It is the manual recovery path for a two-location cluster that has lost a location permanently; see [Recovering From a Lost Location](#recovering-from-a-lost-location). ## Connecting The cluster has no public endpoint: external inbound traffic is closed and no load balancer is created. Clients connect from inside the GVC, subject to [Internal Access](#internal-access), or from your machine through a port-forward. Always use the fully qualified hostname; the bare workload name is not reliably resolvable. | What | Address | Notes | | - | - | - | | PostgreSQL, current primary | `RELEASE_NAME-postgres-proxy.GVC_NAME.cpln.local:5432` | The address to give applications. Every location's proxy reaches the same primary, wherever it is. | | PostgreSQL, pooled | `RELEASE_NAME-postgres-pgbouncer.GVC_NAME.cpln.local:5432` | Only when `pgbouncer.enabled: true`. Prefer it when the pooler is on. | | One specific member | `replica-INDEX.RELEASE_NAME-postgres.LOCATION.GVC_NAME.cpln.local:5432` | `LOCATION` is a name from `global.locations`, `INDEX` is `0` to `replicas - 1`. Useful for read-only queries against a nearby replica; a client using these has to discover which member is the primary itself. | | Patroni REST API | Port `8008` on the same per-member names | `GET /primary` answers `200` only on the leader, `/replica` only on a replica, `/health` on any running member. | | HAProxy health and stats | `RELEASE_NAME-postgres-proxy.GVC_NAME.cpln.local:8404/healthz` and `:8405/stats` | `/healthz` is healthy while a primary backend is reachable. | | Credentials | `cpln secret reveal POSTGRES_CREDENTIALS_SECRET_NAME -o yaml` | The `username`, `password` and `database` keys. Never stored in the Helm release. | To verify from your own machine, tunnel to the proxy and connect with `psql`: ```bash theme={null} cpln port-forward RELEASE_NAME-postgres-proxy 5432:5432 --gvc GVC_NAME ``` ```bash theme={null} psql "postgresql://USERNAME:PASSWORD@127.0.0.1:5432/DATABASE" -c 'select pg_is_in_recovery()' ``` `pg_is_in_recovery()` returns `f` — the proxy always lands on the primary. To see every member, its location, role and replication lag, run `patronictl` inside any member (the startup script writes Patroni's config to `/tmp/patroni_config.yml`): ```bash theme={null} cpln workload exec RELEASE_NAME-postgres --gvc GVC_NAME --location LOCATION_NAME --container patroni-postgres \ -- patronictl -c /tmp/patroni_config.yml list ``` ## Operations ### Backing Up Backups are off by default. Enable them with `backup.enabled: true`, pick a `backup.mode` and a `backup.provider`, and complete that provider's setup below first. Both modes write to the same bucket settings. * **`logical`** — On every run of `backup.logical.schedule` the cron workload connects through the proxy to the current primary, runs `pg_dumpall`, and uploads `postgres-TIMESTAMP.sql.gz` under `backup.PROVIDER.prefix`. The dump includes every database and role. * **`wal-g`** — Every member archives WAL segments with `wal-g wal-push` and, on the member holding the leader lock, the `wal-g-backup` sidecar pushes a base backup every `backup.walg.intervalSeconds` (retrying every minute until the first one succeeds, so a newly promoted leader backs up promptly). Base backups and WAL land under `backup.PROVIDER.prefix`, compressed with `zstd`. **A nearly empty backup object is a failed run, not a backup.** If `pg_dumpall` cannot reach the cluster, the upload still writes an empty gzip under a normal-looking timestamped name and the job exits non-zero. Check the object size before relying on it — a real dump is kilobytes at minimum. After enabling `wal-g`, confirm `SHOW archive_mode` reports `on` in every location before relying on the archive. The AWS S3, GCS and MinIO paths have each been exercised in both modes during the template's test cycles, at the shipped settings. A WAL-G restore into a running release has not — see [Restoring a Backup](#restoring-a-backup). Rehearse your restore on a test release before you need it. #### AWS S3 1. Create an S3 bucket. Set `backup.aws.bucket` to its name and `backup.aws.region` to its region. 2. If you do not have one yet, create a Control Plane [cloud account](/guides/create-cloud-account) for the AWS account holding the bucket. Set `backup.aws.cloudAccountName` to its name. 3. Create an IAM policy scoped to exactly that bucket, replacing `YOUR_BUCKET_NAME`: ```json theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListBucket", "s3:GetObjectVersion", "s3:DeleteObjectVersion" ], "Resource": [ "arn:aws:s3:::YOUR_BUCKET_NAME", "arn:aws:s3:::YOUR_BUCKET_NAME/*" ] } ] } ``` 4. Set `backup.aws.policyName` to that policy's name. The template attaches it, together with `cpln-connector`, to the workload identity and nothing else — the bucket in your policy is the only storage the backup can reach. #### GCS 1. Create a GCS bucket. Set `backup.gcp.bucket` to its name. 2. Create a Control Plane [cloud account](/guides/create-cloud-account) for the GCP project holding the bucket, and set `backup.gcp.cloudAccountName` to its name. 3. Grant the cloud account's service account the **Storage Admin** (`roles/storage.admin`) role on the project. The template additionally binds the identity to `roles/storage.objectAdmin` on exactly the bucket in `backup.gcp.bucket`. #### MinIO No cloud account is needed — the backup authenticates with a second prerequisite secret. 1. Create the bucket in MinIO. Set `backup.minio.bucket` to its name. 2. Set `backup.minio.endpoint` to the S3 API address including the port. For the [MinIO](/template-catalog/templates/minio) template deployed in the same GVC that is `http://WORKLOAD_NAME.GVC_NAME.cpln.local:9000` — use the fully qualified name, because a bare short name does not resolve for every workload type. 3. Create a [dictionary secret](/guides/create-secret/dictionary) holding exactly the keys `accessKey` and `secretKey`, and set `backup.minio.credentialsSecretName` to its name. For the MinIO template these are its admin username and password: ```bash theme={null} cpln secret create-dictionary --name MINIO_CREDENTIALS_SECRET_NAME \ --entry accessKey=MINIO_ACCESS_KEY \ --entry secretKey=MINIO_SECRET_KEY ``` The policy this template creates grants the identity `reveal` on this secret as well as the database credentials secret, and on nothing else. **In `wal-g` mode the endpoint must be reachable from every location.** WAL-G's `restore_command` runs on every member, and a member that cannot reach the archive never finishes starting — it is restarted indefinitely while the leader stays healthy and writes keep succeeding, so the cluster quietly loses its redundancy. A MinIO workload that exists in only one location is exactly that shape. For `wal-g`, point the endpoint at a MinIO deployment present in every location or at an external S3-compatible service; `logical` mode is unaffected because its cron runs only in `backup.location`, and S3 and GCS are global endpoints. #### Manual Dump on Demand For a one-off copy outside the schedule, dump a database from inside a member. Pass the output through `base64` on both sides — the exec stream is text-only and corrupts raw binary — and run `pg_dump` against the local member, which works on a replica as well as on the primary: ```bash theme={null} cpln workload exec RELEASE_NAME-postgres --gvc GVC_NAME --location LOCATION_NAME --container patroni-postgres --quiet -- \ sh -c 'PGPASSWORD="$PG_PASSWORD" pg_dump -Fc -h 127.0.0.1 -U "$PG_USERNAME" "$PG_DATABASE" | base64' \ | base64 -d > DATABASE.pgc ``` The credentials are already present in the container as `PG_USERNAME`, `PG_PASSWORD` and `PG_DATABASE`. ### Restoring a Backup #### Logical Each scheduled backup is a gzip-compressed `pg_dumpall` script, so it is restored by connecting **through the proxy** — which always writes to the current primary — to the `postgres` maintenance database as the superuser from your credentials secret; the script recreates the roles and databases it contains. Run the commands from a client that can reach both the bucket and the cluster: a workload inside the GVC, or your own machine with `cpln port-forward RELEASE_NAME-postgres-proxy 5432:5432 --gvc GVC_NAME` open and `--host=127.0.0.1` in place of the internal hostname. ```bash theme={null} export PGPASSWORD="PASSWORD" aws s3 cp "s3://BUCKET_NAME/PREFIX/postgres-TIMESTAMP.sql.gz" - \ | gunzip \ | psql \ --host=RELEASE_NAME-postgres-proxy.GVC_NAME.cpln.local \ --port=5432 \ --username=USERNAME \ --dbname=postgres unset PGPASSWORD ``` ```bash theme={null} export PGPASSWORD="PASSWORD" gsutil cp "gs://BUCKET_NAME/PREFIX/postgres-TIMESTAMP.sql.gz" - \ | gunzip \ | psql \ --host=RELEASE_NAME-postgres-proxy.GVC_NAME.cpln.local \ --port=5432 \ --username=USERNAME \ --dbname=postgres unset PGPASSWORD ``` ```bash theme={null} export PGPASSWORD="PASSWORD" export AWS_ACCESS_KEY_ID="MINIO_ACCESS_KEY" export AWS_SECRET_ACCESS_KEY="MINIO_SECRET_KEY" aws configure set default.s3.addressing_style path aws s3 cp "s3://BUCKET_NAME/PREFIX/postgres-TIMESTAMP.sql.gz" - --endpoint-url "http://MINIO_ENDPOINT:9000" \ | gunzip \ | psql \ --host=RELEASE_NAME-postgres-proxy.GVC_NAME.cpln.local \ --port=5432 \ --username=USERNAME \ --dbname=postgres unset PGPASSWORD ``` `already exists` errors for the superuser, the `repl` role and the database that the cluster created at bootstrap are expected; the rest of the script loads normally. Restore into an empty cluster (a fresh install, or one whose databases you have dropped) to avoid duplicate-object errors on existing tables. Replicas receive the restored data through streaming replication; nothing has to be done per location. A manual `pg_dump -Fc` file from [Manual Dump on Demand](#manual-dump-on-demand) goes back in with `pg_restore`, again through the proxy and again base64-encoded, with `--stdin` so the file actually reaches the container: ```bash theme={null} base64 DATABASE.pgc | cpln workload exec RELEASE_NAME-postgres --gvc GVC_NAME --location LOCATION_NAME --container patroni-postgres --stdin --quiet -- \ sh -c 'base64 -d | PGPASSWORD="$PG_PASSWORD" pg_restore -h RELEASE_NAME-postgres-proxy.GVC_NAME.cpln.local -U "$PG_USERNAME" -d "$PG_DATABASE" --clean --if-exists' ``` A `pg_dumpall` dump of this template was replayed into a fresh cluster with zero errors during the template's test cycle; the exact commands above follow that procedure but were not themselves run against a 2.0.0 release. Rehearse them on a test release. #### WAL-G **The template provides no supported way to restore a WAL-G backup into the running cluster.** A WAL-G restore needs an empty data directory, the Patroni workload's volume set is named by the release and rendered into the workload, and nothing in `values.yaml` names or substitutes a volume set; re-pointing the workload by hand would be reverted by the next upgrade. What the archive does give you is a complete, restorable copy of the database outside this release. To see what is in the archive, list the base backups from the sidecar, which already carries the bucket settings and credentials: ```bash theme={null} cpln workload exec RELEASE_NAME-postgres --gvc GVC_NAME --location LOCATION_NAME --container wal-g-backup \ -- wal-g backup-list ``` To recover data, fetch a base backup into an **empty** data directory on a scratch PostgreSQL 17 instance outside this release that has the same `WALG_*` and cloud settings as the sidecar — `wal-g backup-fetch DATA_DIRECTORY BACKUP_NAME`, then start PostgreSQL with `restore_command = 'wal-g wal-fetch %f %p'` and a `recovery.signal` file so it replays the archived WAL — and load what you need into the cluster with `pg_dump` / `pg_restore` through the proxy. Follow the [WAL-G documentation](https://wal-g.readthedocs.io/PostgreSQL/) for the restore itself. This WAL-G recovery path is **unverified on this template**: fetching a base backup and replaying WAL into an empty directory was proven during an earlier test cycle, but no end-to-end restore into a running or new release has been run. If you ever reinitialize the cluster (uninstall and reinstall), change `backup.PROVIDER.prefix` before enabling `wal-g` again, or the new cluster's WAL collides with the old system identifier in the same archive. ### Rotating the Password The `password` key is applied when the cluster first bootstraps and is then used in three places: by the superuser role, by the `repl` replication role, and by Patroni's own connections to each member's PostgreSQL. Changing the secret alone changes none of them — a running replica resolves the secret only when it starts. **This rotation has not been exercised on this template**; treat it as a short planned outage and rehearse it on a test release first. Connect through the proxy as the superuser and run: ```sql theme={null} ALTER ROLE USERNAME WITH PASSWORD 'NEW-STRONG-PASSWORD'; ALTER ROLE repl WITH PASSWORD 'NEW-STRONG-PASSWORD'; ``` Edit the `password` key and apply the whole secret manifest: ```bash theme={null} cpln secret reveal POSTGRES_CREDENTIALS_SECRET_NAME -o yaml > credentials.yaml # edit data.password in credentials.yaml, then cpln apply -f credentials.yaml ``` Replicas pick up a changed secret only when they restart. Redeploy the Patroni workload immediately — until its members restart, Patroni and the replicas are still using the old password — and PgBouncer if it is enabled. The backup cron reads the secret fresh on its next run. ```bash theme={null} cpln workload force-redeployment RELEASE_NAME-postgres --gvc GVC_NAME cpln workload force-redeployment RELEASE_NAME-postgres-pgbouncer --gvc GVC_NAME ``` ### Managing the Cluster With patronictl `patronictl` is in the image and reads the config the startup script writes to `/tmp/patroni_config.yml`. Run it inside any member; `--location` picks the member, and any name from `global.locations` will do. Members are named `RELEASE_NAME-postgres-LOCATION-INDEX`. ```bash theme={null} # Every member, its location, role, timeline and replication lag cpln workload exec RELEASE_NAME-postgres --gvc GVC_NAME --location LOCATION_NAME --container patroni-postgres \ -- patronictl -c /tmp/patroni_config.yml list # Move the primary to another location — a planned handover with a brief write pause cpln workload exec RELEASE_NAME-postgres --gvc GVC_NAME --location LOCATION_NAME --container patroni-postgres \ -- patronictl -c /tmp/patroni_config.yml switchover --candidate RELEASE_NAME-postgres-LOCATION-0 --force # The cluster-level settings this cluster is actually running cpln workload exec RELEASE_NAME-postgres --gvc GVC_NAME --location LOCATION_NAME --container patroni-postgres \ -- patronictl -c /tmp/patroni_config.yml show-config ``` The cluster-level Patroni settings are written once, when the cluster is first initialized, and are deliberately not values — a knob would look adjustable while only ever applying to a brand-new cluster. The shipped values are `ttl: 45`, `loop_wait: 10`, `retry_timeout: 15`, `maximum_lag_on_failover: 33554432` (32 MiB), `failsafe_mode: true` and `use_pg_rewind: true`. Change them on a live cluster with `edit-config`, keeping all three timers together because Patroni requires `loop_wait + 2 × retry_timeout <= ttl` and silently substitutes its own values when that is violated: ```bash theme={null} cpln workload exec RELEASE_NAME-postgres --gvc GVC_NAME --location LOCATION_NAME --container patroni-postgres \ -- patronictl -c /tmp/patroni_config.yml edit-config --force \ -s ttl=45 -s loop_wait=10 -s retry_timeout=15 ``` ### Upgrading From 1.0.0 or 1.0.1 Version 1.0.2 dropped the `-ml` infix from every PostgreSQL-side resource name: the workloads, secrets, identity, policy and the **volume set** changed from `…-postgres-ml…` to `…-postgres…` (for example the proxy workload from `…-postgres-ml-proxy` to `RELEASE_NAME-postgres-proxy`, and the data volume set from `…-postgres-ml-vs` to `RELEASE_NAME-postgres-vs`). The etcd resources did not change. **There is no in-place upgrade path across that rename.** A `cpln helm upgrade` from 1.0.0 or 1.0.1 binds the Patroni workload to a new, empty volume set under the new name, and the old proxy hostname disappears. Do not rely on the old volume set surviving; treat the upgrade as a reinstall. Back up first, uninstall, install the new version, restore the dump through the new proxy, and move every application connection string to `RELEASE_NAME-postgres-proxy`. Since every 1.x release also created its own GVC, a cluster on 1.0.0 or 1.0.1 should go straight to 2.0.0 by the procedure below rather than stopping at 1.0.2 — it is the same backup-and-restore either way. ### Migrating From Version 1 Every release before 2.0.0 created its own GVC; 2.0.0 deploys into one you already have. **Do not `cpln helm upgrade` a 1.x release onto 2.0.0.** Once the chart stops declaring a GVC, Helm removes the one the old release created — and deleting a GVC deletes every workload, volume set and identity inside it, including the volumes holding your database. The chart refuses to render when your values still carry the 1.x `global.gvc` key, so a values-carrying upgrade fails safely before any resource is touched; that guard cannot fire on an upgrade run with no values at all. The procedure below is the safety; the render guard is only a backstop. Install 2.0.0 as a **new release against an existing GVC**, move the data across, then remove the old release. Use `backup.mode: logical`, or run `pg_dumpall` through the old release's proxy endpoint (on 1.0.0 and 1.0.1 that workload carries the `-ml` infix). Confirm the object is a real dump and not a nearly empty one before going further — it is the only copy that crosses the boundary. Pick a GVC whose locations are the ones you want members in, and read them back with `cpln gvc get GVC_NAME -o yaml`. Do **not** use the GVC the 1.x release created — it is still owned by the old Helm release and goes away when you uninstall it. In your values, delete `global.gvc.name` and rename `global.gvc.locations` to a top-level `global.locations`. Follow [Installation](#installation), pointing `postgres.credentialsSecretName` at a secret holding the credentials your applications already use. Load the dump through the new release's proxy as shown in [Restoring a Backup](#restoring-a-backup), then repoint every application at `RELEASE_NAME-postgres-proxy.GVC_NAME.cpln.local:5432` (or the PgBouncer address). ```bash theme={null} cpln helm uninstall OLD_RELEASE_NAME --gvc OLD_GVC_NAME ``` Run it against the GVC the old release was *installed into*. Uninstalling removes the release **and the GVC it created**, taking the old volume sets with it — do this only once the restore is verified. Values that changed: | 1.x | 2.0.0 | | - | - | | `global.gvc.name` | Removed — the GVC is wherever you install | | `global.gvc.locations` | `global.locations` | | Bundled etcd Multi-Location 1.x | Pinned to **2.0.0**, which also creates no GVC | ### Scaling and Availability What the cluster survives follows from the quorum table in [Locations and Quorum](#locations-and-quorum); the points below are how each case plays out. * **Losing a replica, or a whole non-primary location, is survived automatically.** The primary keeps serving writes; a member that comes back with its volume intact rejoins and catches up by streaming. The primary retains up to 10 GB of WAL for an absent member (`max_slot_wal_keep_size`) — a durability-for-availability trade, so that a location down for hours cannot fill the primary's volume — and a member absent longer than that re-clones from the primary automatically. * **Losing the primary, or its location, with three or more locations is an automatic failover.** Patroni promotes a replica in a surviving location, every location's proxy re-routes to it, and PgBouncer follows the proxy, so clients need no reconfiguration. The old primary rejoins as a replica when it returns (`use_pg_rewind` is on). Only replicas within `maximum_lag_on_failover` (32 MiB) of the primary are eligible — check the lag column of `patronictl list`. * **Replication is asynchronous, so a failover can lose the most recent transactions** — those that had not yet reached the promoted replica. Check lag on the primary with `pg_stat_replication`, using `application_name` (the member name) to tell members apart. * **With two locations there is no automatic failover.** The survivor holds current data but stays read-only until you promote it by hand — see [Recovering From a Lost Location](#recovering-from-a-lost-location). This manual path has not been exercised on this template. * **A planned move is nearly seamless.** `patronictl switchover` (see [Managing the Cluster With patronictl](#managing-the-cluster-with-patronictl)) hands the leader lock over with a brief write pause, and a member that is shut down cleanly — a restart, a redeploy — performs the same switchover first through its pre-stop hook. * **A `cpln helm upgrade` can restart members in every location together, including an upgrade that changes nothing.** The data survives, but expect a brief write outage while a leader is re-elected, so plan every upgrade — and every change to `primaryLocation`, `global.locations` or `backup.mode` — as a short planned outage with application retries in place. * **A restart of the etcd tier briefly interrupts writes and recovers on its own.** Patroni's failsafe mode lets the primary keep serving while the consensus store is unreachable as long as it can still reach every other member; the cluster re-promotes and resumes without operator action. * **Applications must retry transient errors.** During a failover expect connection-level failures (`server closed the connection unexpectedly`, `terminating connection due to administrator command`) and, briefly, `ERROR: cannot execute INSERT in a read-only transaction` while the proxy's health check catches up with a demoted member. Exponential backoff over a few seconds covers a failover; an upgrade needs a window of a couple of minutes. * **To scale**, raise `global.locations[].replicas` for more members in a location (new members clone from the primary), `proxy.*Replicas` / `pgbouncer.*Replicas` for more capacity per location, and `volumeset.autoscaling` for storage. Do not run more than one etcd member per location — `replicas` affects Patroni only. * **Cost scales with write volume multiplied by the members outside the primary's location.** Each of them receives a full copy of the WAL stream, and cross-region traffic is billed. Read-mostly workloads are inexpensive to stretch; write-heavy ones are not. ### Recovering From a Lost Location With **three or more locations** this section does not apply — losing one location is an automatic failover, and a member that returns rejoins on its own. With **two locations**, losing one loses consensus quorum permanently: the surviving member holds current data but cannot be granted the leader lock, and consensus writes hang rather than failing fast. Recovery is the bundled etcd chart's emergency path, driven from this chart's `etcd.recovery.forceNewClusterInLocation` value. It has not been exercised on this template; follow it on a test release before relying on it. If the location may return, wait for it instead. A member from a location that comes back with its volumes intact rejoins without any of this. Set `etcd.recovery.forceNewClusterInLocation` to the **surviving** location's name and upgrade the release with your existing values: ```bash theme={null} cpln helm upgrade RELEASE_NAME oci://ghcr.io/controlplane-com/templates/postgres-multi-location \ --version 2.0.0 --gvc GVC_NAME -f postgres-locations.yaml \ --set postgres.credentialsSecretName=POSTGRES_CREDENTIALS_SECRET_NAME \ --set etcd.recovery.forceNewClusterInLocation=LOCATION_NAME ``` The surviving etcd member restarts as a single-member cluster rebuilt from its own write-ahead log; once it accepts writes again, Patroni can take the leader lock and promote the surviving PostgreSQL member. Confirm with `patronictl list`. Set `etcd.recovery.forceNewClusterInLocation` back to `""` and upgrade again. Leaving it set means the flag fires on every future restart of that member. A member from the lost location still holds the old consensus cluster ID and refuses to start until its data directory is cleared; the PostgreSQL member there re-clones from the new primary once it can start. Remove the location from `global.locations` and the GVC, or reset its volumes, before bringing it back. Never set `etcd.recovery.forceNewClusterInLocation` to more than one location, and never leave it set. The render-time check only compares the value against `global.locations`, not against the GVC — naming a location the GVC does not have passes validation, no member runs there, and the flag is never applied. ### Troubleshooting When a member will not come up, its own `[patroni]` log lines name the reason (the etcd members log `[etcd]` lines in the same way): ```bash theme={null} cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-postgres"} |= "[patroni]"' --limit 50 --since 10m ``` **Symptom:** the install succeeds, `RELEASE_NAME-postgres` stays not ready in every location indefinitely, and `cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-postgres"}' --limit 50 --since 10m` returns zero lines. **Cause:** `postgres.credentialsSecretName` (or, with MinIO backups, `backup.minio.credentialsSecretName`) names a secret that does not exist, so the container is never started. The message is only visible in `status.versions[].message`: ```bash theme={null} cpln workload get-deployments RELEASE_NAME-postgres --gvc GVC_NAME -o yaml ``` ```text theme={null} The secret POSTGRES_CREDENTIALS_SECRET_NAME no longer exists. Workload updates are paused until the secret is added or the reference to the secret removed. ``` **Fix:** create the secret as in [Prerequisites](#prerequisites). The deployment recovers on its own within several minutes, or run `cpln workload force-redeployment RELEASE_NAME-postgres --gvc GVC_NAME`. **Cause:** your values still carry the 1.x `global.gvc` block. Version 2.0.0 no longer creates a GVC and refuses to render rather than let an in-place upgrade delete the one the old release created. Because the bundled etcd chart renders first, the message may name `etcd-multi-location` rather than this chart; the remedy is the same. **Fix:** do not upgrade in place. Follow [Migrating From Version 1](#migrating-from-version-1): install 2.0.0 as a new release with `global.locations`, restore, then uninstall the old release. **Cause:** the template validates `global.locations` at render time and rejects a list with fewer than two entries, a duplicated location, or a `replicas` below `1`. A `--set global.locations[0].name=...` override replaces the whole list with a single entry and triggers this too. The message may name `etcd-multi-location`, which renders first. **Fix:** supply the full list in a values file (see [Installation](#installation)). For a single-location cluster use [PostgreSQL Highly Available](/template-catalog/templates/postgres-highly-available). **Cause:** `backup.mode: logical` with `proxy.enabled: false` and `pgbouncer.enabled: false`. The dump must run against the current primary, and without the proxy there is no stable address for it. **Fix:** set `proxy.enabled: true` (enabling PgBouncer also enables it), or use `backup.mode: wal-g`, which runs on the members themselves. **Symptom:** a freshly installed member exits with code `1`; the log names the missing location(s) and reads `That is a bare majority: the cluster would form, report healthy, and have ZERO fault tolerance`. **Cause:** `global.locations` names a location the GVC does not have. The members in the other locations could still form a cluster, but one with no fault tolerance, so a fresh member refuses to bootstrap it. An already-initialized member logs the same condition as a `[patroni] WARNING` and keeps serving. **Fix:** add the missing location to the GVC, or remove it from `global.locations`, and redeploy. `cpln gvc get GVC_NAME -o yaml` shows the GVC's list under `spec.staticPlacement.locationLinks`. **Cause:** fewer than a majority of the locations in `global.locations` exist in the GVC, so the consensus store can never elect a leader and Patroni could never promote a primary. A fresh member exits immediately rather than waiting forever; an already-initialized member logs a `[patroni] WARNING` and keeps running. **Fix:** add the missing locations to the GVC or remove them from `global.locations` so that the GVC has every declared location, and redeploy. **Cause:** a replica started in a GVC location that `global.locations` does not list. Such a member would join the cluster through etcd and could even win the leader election while no proxy routes to it, so it exits unconditionally. Normally an unlisted GVC location runs nothing at all; this line only appears if a replica was placed there anyway. **Fix:** add that location to `global.locations` and upgrade, or remove it from the GVC. **Cause:** the member could not read its own GVC at boot — the API was briefly unavailable, or the `RELEASE_NAME-postgres-gvc-policy` policy is missing — so it skipped the GVC location check and started anyway. The warning is informational; the member runs normally. **Fix:** nothing, if the policy exists and the warning was transient. If it recurs on every start, confirm the policy with `cpln policy get RELEASE_NAME-postgres-gvc-policy -o yaml` and that it grants `view` on the GVC the release is installed into. **Cause:** `primaryLocation` is set, but no member in that location initialized the cluster within the 90-second head start — it was slow, down, or (as the same boot check will have warned) names a location the GVC does not have. The member that logged this bootstrapped the cluster where it is. The cluster is healthy; only the primary's placement differs from your preference. **Fix:** once every member is up, move the primary with `patronictl switchover --candidate RELEASE_NAME-postgres-LOCATION-0 --force` as shown in [Managing the Cluster With patronictl](#managing-the-cluster-with-patronictl). Do not change `primaryLocation` to force it — that restarts every member. **Cause:** almost always arithmetic rather than a fault. With two locations the consensus store cannot grant the leader lock after losing either one (see [Locations and Quorum](#locations-and-quorum)). With three or more, a replica lagging more than `maximum_lag_on_failover` (32 MiB) is excluded from the election. **Fix:** run `patronictl list` and read the `Lag in MB` column. With two locations follow [Recovering From a Lost Location](#recovering-from-a-lost-location); with three or more, wait for the lagging replica to catch up, or check `pg_stat_replication` on the primary if it never does. **Cause:** the consensus store cannot accept writes — the etcd cluster has lost quorum, or its backend quota is full and it has raised a `NOSPACE` alarm — so Patroni cannot renew its leader lease and exits cleanly. The restarts look healthy because the exit code is zero. **Fix:** check etcd before touching the database: `cpln workload get-deployments RELEASE_NAME-etcd --gvc GVC_NAME -o yaml`, then the read-only `etcdctl` commands in the etcd Multi-Location page's [If the Backend Quota Is Already Full](/template-catalog/templates/etcd-multi-location#if-the-backend-quota-is-already-full) section. **Cause:** before starting HAProxy, the proxy waits until every Patroni member in every location answers on port `8008`, naming the member it is waiting for. One that never answers — a member wedged on a missing secret, or a declared location the GVC does not have — blocks every proxy in every location. After the gate's timeout the proxy container exits and is restarted to check again. **Fix:** fix the named member first (the accordions above); the proxies start on their own once every member answers. **Cause:** `pg_dumpall` failed — usually because the job could not reach the proxy or authenticate — and the upload pipeline still wrote an empty gzip under a normal-looking name; the job itself reported `failed`. **Fix:** read the job's log with `cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-postgres-backup"}' --limit 50 --since 24h`, fix the cause, and treat the object as not a backup. **Cause:** the backup container ran out of memory. This is what the GCS path does at `128Mi`: a `logical` job dies without an error line and the `wal-g-backup` sidecar is killed in a loop, taking its member out of service while WAL keeps archiving with no base backup. `cpln workload get-deployments RELEASE_NAME-postgres --gvc GVC_NAME -o yaml` shows the restart reason. **Fix:** keep `backup.resources.memory` at the shipped `512Mi` or higher. **Cause:** WAL-G's `restore_command` runs on every member, and the MinIO endpoint is reachable only from the location it runs in. Members elsewhere cannot fetch from the archive and are restarted indefinitely while the leader stays healthy and writes keep succeeding. **Fix:** point `backup.minio.endpoint` at a MinIO deployment present in every location or at an external S3-compatible service, or use S3 or GCS. Disabling backups heals the members. **Cause:** `pgbouncer.maxDbConnections` is enforced per PgBouncer replica, so the tier can open up to `maxReplicas × maxDbConnections` server connections — 400 at the shipped values — against the cluster's `max_connections` of 100. **Fix:** lower `pgbouncer.maxDbConnections` (and `defaultPoolSize`) so that `maxReplicas × maxDbConnections` stays comfortably under 97. **Cause:** a failover or switchover is in progress. The proxy's health check has not yet noticed that the member it was forwarding to has been demoted, so a few connections land on a read-only replica. **Fix:** nothing on the cluster — it clears as soon as the health check flips to the new primary. Make sure applications retry transient errors with backoff, as described in [Scaling and Availability](#scaling-and-availability). **Cause:** the secret is applied only when the cluster first bootstraps. Afterwards the stored password lives in the database, not the secret, and running members resolve the secret only when they start. **Fix:** change it inside PostgreSQL and then update the secret and redeploy the consumers — see [Rotating the Password](#rotating-the-password). ## Important Notes * **Do not `cpln helm upgrade` a 1.x release onto 2.0.0.** It deletes the GVC the old release created and every volume set in it. Follow [Migrating From Version 1](#migrating-from-version-1): new release, restore, then uninstall the old one. * **Create the credentials secret before installing.** A reference to a secret that does not exist wedges the workload in every location with no log output; see [Prerequisites](#prerequisites) for the one command that shows the reason. * **`global.locations` must match the GVC you install into.** A declared location the GVC lacks makes a fresh member refuse to start with a named `[patroni] FATAL` line; a GVC location you did not list runs nothing. Read the GVC's list with `cpln gvc get GVC_NAME -o yaml` before installing. * **Quorum arithmetic decides your failure tolerance.** Two locations survive zero losses and promotion is manual; three give automatic failover. * **Set `primaryLocation` before the first install if you care where the primary is.** Changing it later restarts every member to move the leader; use `patronictl switchover` instead. * **Credentials are read only when the cluster first bootstraps.** Rotate with `ALTER ROLE` on the superuser and `repl`, then update the secret and redeploy — see [Rotating the Password](#rotating-the-password). * **Replication is asynchronous.** A failover can lose the most recent transactions, bounded by the replication lag at the moment of failure. * **Every `cpln helm upgrade` can restart members in every location together, including a no-op one.** Expect a brief write outage; the data survives. The same applies to changing `primaryLocation`, `global.locations` or `backup.mode`. * **With `internalAccess.type: workload-list`, list only your clients** — this release's own tiers are added for you. The bundled etcd tier is separate: `etcd.internalAccess.type: workload-list` requires adding `//gvc/GVC_NAME/workload/RELEASE_NAME-postgres` by hand. * **Leave etcd compaction at its defaults.** An uncompacted consensus store goes read-only and takes the database's failover with it. * **A WAL-G backup cannot be restored into the running cluster through this template.** The archive is restorable outside the release; see [Restoring a Backup](#restoring-a-backup). * **`cpln helm uninstall` deletes the volume sets**, and the database with them, after a final snapshot kept 7 days. Your credentials secret is yours and is left alone. * **Cost scales with write volume multiplied by the members outside the primary's location.** Cross-region replication traffic is billed. ## External References Patroni clustering and automatic failover documentation Command reference for inspecting, switching over and reconfiguring a cluster Continuous archiving, base backups and restore Official PostgreSQL documentation PgBouncer configuration reference View the source files, default values, and chart definition