pgEdge - Control Plane

Overview

pgEdge is an active-active distributed PostgreSQL cluster built on the Spock multi-master replication extension. Every node accepts reads and writes, and a row written on any node replicates to all the others with last-update-wins conflict resolution. This template deploys one set of PostgreSQL 17 nodes in each location you list, a per-location HAProxy failover tier that keeps a location serving when one of its nodes fails, a per-location PgBouncer connection pooler that gives applications a single endpoint, and an optional scheduled backup to S3 or GCS that also restores with one command. Database credentials are not template values. The nodes, the pooler and the backup job read the username, password and database name from a dictionary secret you create before installing, so no password passes through Helm or lands in the release.

This template deploys into an existing GVC that you already have; it does not create, provision or manage one. Every location you list in locations must already be one of that GVC’s locations. A GVC location you do not list starts nothing — its deployment reads This workload location is deactivated because maxScale is set to 0., which is what a healthy install looks like in a shared GVC, not a fault.

What Gets Created

Resource	Name	Purpose	Optional
Workload	`RELEASE_NAME-pgedge`	The PostgreSQL 17 + Spock nodes (stateful, `locations[].replicas` nodes in each listed location, TCP `5432`). Every node is writable and individually addressable	—
Workload	`RELEASE_NAME-pgedge-proxy`	Per-location HAProxy failover tier (standard, `proxy.minReplicas` to `proxy.maxReplicas` per location, TCP `5432`, plus HTTP `8404` health and `8405` stats). Each location’s PgBouncer connects to it	—
Workload	`RELEASE_NAME-pgbouncer`	PgBouncer connection pooler, the endpoint applications connect to (standard, `pgbouncer.minReplicas` to `pgbouncer.maxReplicas` per location, autoscales on requests per second, TCP `5432`)	—
Workload	`RELEASE_NAME-pgedge-backup`	Scheduled `pg_dump` of your database to S3 or GCS, and the one-command restore (cron). Runs only in the first entry of `locations`	`backup.enabled: true`
Volume Set	`RELEASE_NAME-pgedge-vs`	One `ext4` general-purpose SSD volume per node. Snapshotted on `volumeset.snapshots.schedule` (daily at 03:00 UTC by default), kept for `volumeset.snapshots.retentionDuration`, with a final snapshot on deletion	—
Identity	`RELEASE_NAME-pgedge-identity`	Shared by every workload in the release; carries the cloud account binding when backups are enabled	—
Policy	`RELEASE_NAME-pgedge-policy`	Grants the identity `reveal` on this release’s secrets and on your credentials secret, and on nothing else	—
Policy	`RELEASE_NAME-pgedge-gvc-policy`	Grants the identity `view` on exactly the one GVC the release is installed into, so each node can confirm at startup that the GVC has every location in `locations`	—
Secret	`RELEASE_NAME-pgedge-startup`	The node startup script (initialization, Spock node and subscription setup)	—
Secret	`RELEASE_NAME-pgedge-config`	The backup destination (bucket and region)	`backup.enabled: true`

Your credentials secret is created by you and is never part of the release.

Prerequisites

Two things must exist before you install: a GVC that already has the locations you intend to use, and the credentials secret. Secrets are org-level, so no GVC flag is involved when creating one.

A GVC that contains every location you will list

The template deploys into the GVC you pass to --gvc. It may have more locations than you list — nothing pgEdge-related runs in those — but it must have every one you do list. Check its locations under spec.staticPlacement.locationLinks:

cpln gvc get GVC_NAME -o yaml

Set locations to a subset of what you see there. A location that is not on the GVC is not caught at install time — the install succeeds and the nodes then refuse to initialize (see Troubleshooting).

Create the database credentials secret

A dictionary secret holding exactly three keys — username, password and database. The nodes create that user and that database on first boot, PgBouncer serves only that database, and its admin console accepts the same username and password. Do not name the database pgbouncer; that name is reserved for the admin console.

cpln secret create-dictionary --name SECRET_NAME \
  --entry username=myuser \
  --entry password='YOUR-STRONG-PASSWORD' \
  --entry database=mydb

Set postgres.credentialsSecretName to the name you used. Secret names are org-wide, so give each release its own.

Read the secret back later

Pass -o yaml. A bare cpln secret reveal prints only a summary table, not the values:

cpln secret reveal SECRET_NAME -o yaml

Create the secret before installing, or the deployment wedges silently. The template refuses to render when postgres.credentialsSecretName is blank, but a name that points at a secret which does not exist installs “successfully” and then never starts. The container never runs, so cpln logs returns zero lines — there is nothing to log. The one place the reason appears is status.versions[].message:

cpln workload get-deployments RELEASE_NAME-pgedge --gvc GVC_NAME -o yaml

Use get-deployments — plain cpln workload get has no versions key. Creating the missing secret repairs the deployment on its own within several minutes, or run cpln workload force-redeployment RELEASE_NAME-pgedge --gvc GVC_NAME to skip the wait.

Backups need a bucket and, for AWS or GCP, a Control Plane Cloud Account before they can be enabled — see Backing Up. Nothing else is required.

Installation

The default locations list names three AWS regions with three nodes each, so a real install nearly always carries its own location list. Lists cannot be set piecemeal with --set — Helm replaces the whole list, and a --set locations[0].name=... without replicas fails to render with location 'aws-us-east-1' must have at least 1 replica — so put them in a values file:

# values.yaml
locations:
  - name: aws-us-east-1
    replicas: 1
postgres:
  credentialsSecretName: SECRET_NAME

cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/pgedge \
  --version 3.0.0 --gvc GVC_NAME \
  -f values.yaml

To install, follow the instructions for your preferred method:

UI

Browse, install, and manage templates visually

CLI

Manage templates from your terminal

Terraform

Declare templates in your Terraform configurations

Pulumi

Declare templates in your Pulumi programs

Configuration

Locations and Replicas

Each entry pairs a location with a node count. Every location listed must already be on the GVC you install into; extra GVC locations run nothing. Listing the same location twice is rejected at render, because duplicates would produce duplicate Spock node names.

locations: # For replicas: use 1 for dev/testing, 3 for production
  - name: aws-us-west-2
    replicas: 3
  - name: aws-us-east-2
    replicas: 3
  - name: aws-eu-central-1
    replicas: 3

multiZone: false  # Set to true to spread replicas across availability zones within each location

Environment	Replicas per location
Dev / testing	1
Production	3

The first entry of locations is special: it is the only location the backup job runs in. Verify that your locations support multiple availability zones before enabling multiZone.

Image and Resources

The node container exposes both a reservation and a limit. maxCpu: 2 against minCpu: 500m is exactly the 4:1 ratio a stateful workload allows, so raise minCpu whenever you raise maxCpu.

image: ghcr.io/pgedge/pgedge-postgres:17-spock5-standard

resources:
  minCpu: 500m
  minMemory: 1Gi
  maxCpu: 2
  maxMemory: 4Gi

Each node accepts 300 connections. The template sets this; it is not a values key.

Credentials

postgres:
  credentialsSecretName: my-pgedge-credentials

The named dictionary secret must exist before install and hold username, password and database — see Prerequisites. The nodes, PgBouncer, HAProxy’s health checks and the backup job all read it; nothing else in the release carries a credential.

Volume Set

Set the initial storage capacity per node (minimum 10 GiB), the volume snapshot schedule, and whether volumes grow as data grows:

volumeset:
  capacity: 10  # Initial capacity in GiB (minimum is 10)
  snapshots:
    schedule: "0 3 * * *"    # volume snapshots of every node, daily at 3am UTC; at most hourly; "" turns them off
    retentionDuration: 7d    # how long each snapshot is kept
  autoscaling:
    enabled: false  # Set to true to enable autoscaling
    maxCapacity: 100  # Maximum capacity in GiB when autoscaling is enabled
    minFreePercentage: 10  # Minimum free percentage to trigger scaling
    scalingFactor: 1.2  # How much to scale up when triggered

snapshots.schedule must be a cron expression with a single fixed minute; anything more frequent than hourly is refused at render. A final snapshot of each volume is taken when the volume set is deleted, whether or not a schedule is set.

HAProxy Failover Tier

The request flow is app → PgBouncer (pooler) → HAProxy (failover) → pgEdge node. PgBouncer pools connections; HAProxy picks the node. Both run one set per location, and the endpoint applications connect to is the PgBouncer one. Each location’s HAProxy sends all of that location’s traffic to its own first node (replica-0) and, if that node is unhealthy, fails over in order to the other local nodes, then to the nodes of the other locations in locations order — so applications keep serving through a node failure without changing anything. Every location writes to its own node, so the cluster is active/active. The tier is always on.

proxy:
  image: haproxy:3.0.28  # pinned exact; Debian variant (perl needed by the startup check)
  resources:
    cpu: 100m
    memory: 128Mi
  minReplicas: 2         # per location
  maxReplicas: 2         # per location

All of a location’s statements land on whichever node HAProxy currently prefers, so local reads are not spread across a location’s other nodes; that is the deliberate trade for read-your-writes consistency within a location.

PgBouncer Connection Pooler

PgBouncer multiplexes application connections into a smaller pool of real database connections, protecting PostgreSQL from connection exhaustion under high concurrency. It serves only the database named in your credentials secret. minReplicas and maxReplicas are per location, and PgBouncer autoscales between them on request rate.

pgbouncer:
  image: ghcr.io/cloudnative-pg/pgbouncer:1.26.0-202610011121-trixie  # pinned immutable build
  poolMode: transaction  # options: session, transaction, statement
  defaultPoolSize: 20    # real Postgres connections per PgBouncer replica; node max_connections is 300
  maxClientConn: 1000    # client connections accepted per PgBouncer replica
  resources:
    cpu: 500m            # PgBouncer is single-threaded: add replicas rather than cores
    memory: 128Mi
  minReplicas: 2         # per location
  maxReplicas: 4         # per location

Connection budget. Every PgBouncer replica in a location pools onto that location’s replica-0, and each node accepts 300 connections. The defaults use at most 20 × 4 = 80 per location, so one node can carry three locations’ pools after a failover. If you raise defaultPoolSize or maxReplicas, keep defaultPoolSize × maxReplicas × number of locations under about 290.

Pool mode	Behavior
`transaction`	Connection held only for the duration of a transaction. Best for most web and API workloads. Protocol-level prepared statements work; session-level `SET`, advisory locks and `LISTEN` do not. A temporary table works only within a single transaction (`CREATE TEMP TABLE … ON COMMIT DROP`): one created outside a transaction stays on the pooled server connection and is visible to other clients.
`session`	Connection held for the entire client session. Compatible with all PostgreSQL features but provides less connection reuse. A session idle for more than 1 hour is closed by the failover tier; clients reconnect.
`statement`	Connection returned after every statement. Multi-statement transactions are rejected. Rarely used.

Session settings in the connection string. In transaction mode (the default), clients may pass these through the options startup parameter (PGOPTIONS='-c statement_timeout=5s', libpq options=, JDBC options=), and PgBouncer applies them on every server connection the client uses: statement_timeout, lock_timeout, idle_in_transaction_session_timeout, idle_session_timeout, work_mem, maintenance_work_mem, default_transaction_isolation, client_min_messages, plus search_path, application_name, TimeZone, DateStyle and client_encoding. Any other setting in options is refused at connect with unsupported startup parameter in options — set it with ALTER ROLE … SET instead. In session mode, options is reliably applied only on a brand-new server connection; a client handed a pooled connection silently gets the server defaults. Run SET at the start of each session instead, or ALTER ROLE … SET on every node — that reaches new server connections only, so run RECONNECT on the PgBouncer admin console or wait up to 5 minutes.

Internal Access

Controls which workloads can connect to the nodes, the failover tier and PgBouncer. Nothing in this template is exposed publicly, and there is no public access option.

internal_access: # Applies to the pgedge, PgBouncer and HAProxy workloads
  type: same-gvc  # options: same-gvc, same-org, workload-list
  workloads:  # For workload-list: list only your clients --
              # this release's own workloads are added automatically
    #- //gvc/GVC_NAME/workload/WORKLOAD_NAME

same-gvc — Allow access from all workloads in the same GVC.
same-org — Allow access from all workloads in the org.
workload-list — Allow access only from the workloads you list, plus this release’s own workloads, which the template adds for you.

With workload-list, list only your own client workloads. The same list governs Spock replication between the nodes, HAProxy’s and PgBouncer’s connections, and the backup job’s connection, so the template always adds this release’s pgedge, PgBouncer, HAProxy and (when enabled) backup workloads to it. Anything of the release’s you add by hand is de-duplicated.

Backup

When backup.enabled is true, a cron workload dumps your database on the configured schedule and uploads it to S3 or GCS. The same workload performs restores. Keep schedule quoted; any valid cron expression works, including one beginning with *. The bucket, cloud account and IAM setup are described under Backing Up.

backup:
  enabled: false
  image: ghcr.io/controlplane-com/backup-images/postgres-backup:17.1.0
  schedule: "0 2 * * *"   # daily at 2am UTC — runs in locations[0] only
  activeDeadlineSeconds: 21600  # a backup or restore still running after 6 h is stopped

  resources:
    cpu: 100m
    memory: 256Mi  # 128Mi measured OOM-killing the GCS upload

  provider: aws  # Options: aws or gcp

  aws:
    bucket: my-backup-bucket
    region: us-east-1
    cloudAccountName: my-backup-cloudaccount
    policyName: my-backup-policy
    prefix: pgedge/backups  # folder where backups will be stored

  gcp:
    bucket: my-backup-bucket
    cloudAccountName: my-backup-cloudaccount
    prefix: pgedge/backups  # folder where backups will be stored

Connecting

Your application must retry — build it in from the start. Restarts, upgrades and failovers are routine, and each one fails a few connections or statements for a few seconds even though the database stays up. Retry a failed connection with a short backoff (about 0.5–1 seconds) for 10–15 seconds, and retry a failed statement, making retried writes idempotent. Connection pools usually replace a broken connection but do not re-run the failed query for you. Details: Scaling and Availability.

Connect through PgBouncer for all application traffic. The per-location HAProxy behind it handles node failover, and the endpoint does not change. Always use the fully-qualified .GVC_NAME.cpln.local form — the bare workload name does not resolve reliably from every workload type.

What	Address	Notes
Pooled endpoint (use this)	`RELEASE_NAME-pgbouncer.GVC_NAME.cpln.local:5432`	Internal only. Each client reaches the PgBouncer in its own location
Unpooled, with failover	`RELEASE_NAME-pgedge-proxy.GVC_NAME.cpln.local:5432`	HAProxy directly. Counts against the node’s 300 connections
A single node, directly	`replica-N.RELEASE_NAME-pgedge.LOCATION_NAME.GVC_NAME.cpln.local:5432`	Needed for schema changes and per-node checks. `N` runs from `0` to `replicas - 1`. Unpooled
PgBouncer admin console	Pooled endpoint, database `pgbouncer`	Same username and password as your credentials secret
Database, username, password	`cpln secret reveal SECRET_NAME -o yaml`	The `database`, `username` and `password` entries of your credentials secret. The pooler serves only that database
Public endpoint	None	The template has no public access option

PgBouncer does not offer TLS — use sslmode=disable or sslmode=prefer; traffic stays inside the GVC. Clients must support SCRAM-SHA-256 authentication (libpq 10+, JDBC 42.2+). To verify from your own machine, tunnel to PgBouncer and run a query against the tunnel:

cpln port-forward RELEASE_NAME-pgbouncer 5432:5432 --gvc GVC_NAME

psql "postgresql://USERNAME:PASSWORD@127.0.0.1:5432/DATABASE" -c 'select version()'

To check replication from inside a node, run psql on it — the container already carries the credentials in its environment:

cpln workload exec RELEASE_NAME-pgedge --gvc GVC_NAME --replica RELEASE_NAME-pgedge-0 --location LOCATION_NAME --container pgedge -- bash -c 'psql -U "$PG_USERNAME" -d "$PG_DATABASE" -c "SELECT subscription_name, status FROM spock.sub_show_status();"'

On a healthy cluster of N nodes every node returns N−1 rows and each reads replicating.

Schema Changes and DDL

Spock replicates row-level changes (INSERT, UPDATE, DELETE) automatically. DDL does not replicate. A plain CREATE TABLE or ALTER TABLE applies only to the node you ran it on; the other nodes never learn about it, and rows written into that table on one node cannot be applied on a node where it does not exist. Run DDL against a node directly, never through PgBouncer.

Every table must have a PRIMARY KEY. The template adds each new table to the default replication set automatically, and that set replicates UPDATE and DELETE, which Spock cannot do without a key. A table without one does not merely fail to replicate — the CREATE TABLE itself is rejected. Temporary and UNLOGGED tables are exempt: they are never replicated and stay on the node that created them.

ERROR:  table events cannot be added to replication set default
DETAIL:  table does not have PRIMARY KEY and given replication set is configured to replicate UPDATEs and/or DELETEs

Creating a Table

The simplest correct procedure is to run the same CREATE TABLE on every node. The auto-add trigger fires locally on each one, so the table ends up in the default replication set everywhere and DML replicates in all directions:

-- Run on EVERY node, connecting to each directly (not through PgBouncer)
CREATE TABLE orders (
  id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  amount numeric,
  created_at timestamptz DEFAULT now()
);

For larger clusters you can broadcast the DDL instead — but it takes two steps, and the second one runs on the other nodes, not on the node that broadcast:

-- Step 1: on ONE node -- creates the table on all nodes
SELECT spock.replicate_ddl('CREATE TABLE orders (
  id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  amount numeric,
  created_at timestamptz DEFAULT now()
);');

-- Step 2: on every OTHER node -- adds the table to that node's replication set
SELECT spock.repset_add_table('default', 'orders'::regclass);

Step 2 is needed because Spock suppresses event triggers while applying replicated changes, so the auto-add trigger fires only on the node that called replicate_ddl. Running step 2 on that node fails with duplicate key value violates unique constraint "replication_set_table_pkey", and skipping it strands writes made on the other nodes. Confirm on every node — each must return a row:

SELECT * FROM spock.tables WHERE relname = 'orders';

Other DDL

ALTER TABLE and DROP TABLE follow the same rule — apply on every node, or broadcast once:

SELECT spock.replicate_ddl('ALTER TABLE orders ADD COLUMN status text DEFAULT ''pending'';');
SELECT spock.replicate_ddl('DROP TABLE orders;');

Primary Keys

Use uuid primary keys instead of serial or bigserial. Each node maintains its own sequence, so auto-increment integers collide when the same value is generated on two nodes at once:

-- Good: globally unique, no conflicts
id uuid PRIMARY KEY DEFAULT gen_random_uuid()

-- Avoid: causes duplicate key conflicts under concurrent multi-node writes
id serial PRIMARY KEY

Operations

Backing Up

When backup.enabled is true, the RELEASE_NAME-pgedge-backup cron workload dumps your database (the database entry of your credentials secret) with pg_dump in custom format and uploads it as PREFIX/pgedge-RELEASE_NAME-YYYY-MM-DDTHH-MM-SSZ.dump. Every node holds a full copy, so the job dumps the first node that accepts a login and moves on to the next if a dump fails. A run that fails part-way leaves no backup behind. The job runs only in the first location in locations, so backups pause while that whole location is down. Roles and large objects (lo_*) are not included: Spock does not replicate large objects, and the user from your credentials secret owns everything that is restored. To take a backup now, start the job in your first location (the first entry of locations):

cpln workload cron start RELEASE_NAME-pgedge-backup --gvc GVC_NAME --location LOCATION_NAME

AWS S3 Prerequisites

Create your S3 bucket and note its name and region. Set backup.aws.bucket and backup.aws.region.
If you do not have a Cloud Account set up, refer to the docs to Create a Cloud Account. Note the cloud account name.
Create a new AWS IAM policy with the following JSON (replace YOUR_BUCKET_NAME):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:ListBucket",
                "s3:GetObjectVersion",
                "s3:DeleteObjectVersion",
                "s3:GetBucketLocation",
                "s3:AbortMultipartUpload",
                "s3:ListBucketMultipartUploads",
                "s3:ListMultipartUploadParts"
            ],
            "Resource": [
                "arn:aws:s3:::YOUR_BUCKET_NAME",
                "arn:aws:s3:::YOUR_BUCKET_NAME/*"
            ]
        }
    ]
}

Set backup.aws.cloudAccountName to your cloud account name.
Set backup.aws.policyName to the policy created in step 3.

If your IAM policy predates template version 1.1.1, update it to the full action list above. Earlier versions attached the AWS managed policy ReadOnlyAccess to the backup identity as well, which granted read access to every bucket in your account and silently supplied any read action your own policy omitted. The identity now carries cpln-connector and your bucket-scoped policy only.

GCS Prerequisites

Create your GCS bucket and note its name. Set backup.gcp.bucket.
If you do not have a Cloud Account set up, refer to the docs to Create a Cloud Account. Note the cloud account name.
Grant the cloud account’s GCP service account the Storage Admin role. Control Plane uses it to give the backup job roles/storage.objectAdmin on this one bucket.
Set backup.gcp.cloudAccountName to your cloud account name.

Restoring a Backup

The backup job also restores. It puts the schema on every node (DDL does not replicate), adds each table to the replication set, loads the data once so Spock replicates it, sets sequences on every node, waits until every node holds the same row counts, and finally refreshes materialized views on every node. It refuses to run unless every node is reachable and the database is empty on every node, so it can never merge into or overwrite data. Keep applications from writing to the database while it runs. Restore this release’s newest backup:

cpln workload cron start RELEASE_NAME-pgedge-backup --gvc GVC_NAME --location LOCATION_NAME \
  --env PGEDGE_ACTION=restore

Or a specific one — a file name under the prefix, or a full path inside the bucket:

cpln workload cron start RELEASE_NAME-pgedge-backup --gvc GVC_NAME --location LOCATION_NAME \
  --env PGEDGE_ACTION=restore --env RESTORE_FILE=pgedge-RELEASE_NAME-YYYY-MM-DDTHH-MM-SSZ.dump

Follow the run (status successful or failed):

cpln workload cron get RELEASE_NAME-pgedge-backup --gvc GVC_NAME

Read the log for the reason a run failed:

cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-pgedge-backup", container="backup-pgedge"} |= "pgedge-backup"' --since 1h --limit 200

Recovering into a new release (lost cluster, new GVC): install with backup.enabled: true against the same bucket, then restore with RESTORE_FILE=OLD_PREFIX/pgedge-OLD_RELEASE-….dump.
Restoring over an existing database, or retrying a restore that failed part-way: drop your objects (tables, views, sequences, functions, types, non-public schemas) on every node first, connecting to each node directly. Tables in a replication set need CASCADE (DROP TABLE orders CASCADE;). Do not drop the public schema: it holds the template’s auto-replication trigger, which a node recreates only when it is first created.
The archive is downloaded to the job’s local disk before it is applied. For a large database, raise backup.activeDeadlineSeconds.
Ownership and GRANTs are not restored; every object belongs to the credentials user.
Only backups written by template version 3.0.0 or later can be restored this way. The .sql.gz files earlier versions wrote are not read by the job.

Each node’s volume is also snapshotted on volumeset.snapshots.schedule. Restoring a single node’s volume into a running multi-master cluster has not been verified and is not a substitute for the restore above.

Upgrading from 1.x

Template versions through 1.1.1 created their own GVC, so that GVC is part of a 1.x release’s manifest. Version 2.0.0 and later do not declare one — and Helm deletes what a chart stops declaring.

An in-place helm upgrade from 1.x to any later version is not supported and destroys the cluster. The upgrade would delete the GVC the 1.x chart created together with every workload, volume set and identity inside it — including your data. The only supported path is the migration below: install the current version as a new release into an existing GVC, move the data, then remove the old release.The chart refuses to render when your values still carry the 1.x gvc key, so an upgrade that passes your old values file stops before a single API call is made:

Error: execution error at (pgedge/templates/identity.yaml:2:4): pgedge 2.0.0: the `gvc` values key was REMOVED. This chart no longer creates a GVC -- it deploys into the GVC you install into, and `gvc.locations` moved to the top-level `locations`. DO NOT `helm upgrade` a 1.x release onto 2.0.0: the upgrade drops `kind: gvc` from the manifest and Helm deletes what a chart no longer declares, which DESTROYS that GVC and every workload, volumeset and identity inside it. Install 2.0.0 as a NEW release against an existing GVC, move your data, then uninstall the old release. See `Migrating from 1.x` in the README.

The guard cannot cover an upgrade run with no values at all. A 1.x release installed on pure defaults has no gvc key for the chart to see, so nothing fires. Do not run helm upgrade on a 1.x release against any later chart under any circumstances.

The 1.0.x credential values (postgres.username, postgres.password, postgres.database) are refused at render as well, with their own message, so pasting an old values file into a new install produces an error rather than a silent misconfiguration.

Back up the old cluster

Use backup.enabled on the 1.x release, or take a manual pg_dump against one of its nodes directly, at replica-0.OLD_RELEASE-pgedge.LOCATION_NAME.OLD_GVC.cpln.local.If replication on the old cluster was never switched on — which is every 1.x install that did not apply the fix below — the nodes may hold different data. Compare row counts per node before deciding which one to dump, or reconcile them first.

Choose the GVC for the new release

Pick a GVC that already has every location you intend to list in locations. See Prerequisites.

Create the credentials secret and install as a new release

Use a different release name — secret names are org-wide, so a same-named release collides with the 1.x one even in another GVC. Coming from 1.0.2 or earlier, the credentials were plain values with a published default password; choose a new password now and put it in the dictionary secret, since a fresh install creates its roles from that secret.

Load the data and cut over

A 1.x backup is not in the format the restore job reads, so load it into the new cluster by hand, respecting the rules in Schema Changes and DDL: the schema on every node first, then the data on one node only. This manual load is not covered by the restore job, so rehearse it on a scratch release first. Then point your applications at the new pooled endpoint, NEW_RELEASE-pgbouncer.NEW_GVC.cpln.local:5432.

Uninstall the old release against the GVC you installed it into

Not the GVC it created — the GVC you passed at install time is where Helm tracks the release, and uninstalling from there removes the created GVC with it.

Switching Replication On for an Existing 1.x Cluster

PostgreSQL 17.11 ships output_plugin_libraries = 'pgoutput, test_decoding', and Spock’s output plugin is not on that list. On every template version through 1.1.1 this means no replication slot could ever be created: each subscription sits at down, each node accepts writes that never leave it, and every status surface still reports healthy. The reason appears in the node’s own PostgreSQL server log, not in cpln logs:

ERROR:  library "spock_output" may not be used as an output plugin

Version 2.0.0 and later write the setting into postgresql.conf during initdb, so a fresh install needs nothing. An existing data directory keeps the old setting, so if you are staying on 1.x while planning the migration, apply it by hand on every node, connecting to each node directly:

psql "host=replica-0.RELEASE_NAME-pgedge.LOCATION_NAME.GVC_NAME.cpln.local user=USERNAME dbname=DATABASE" \
  -c "ALTER SYSTEM SET output_plugin_libraries = pgoutput, test_decoding, spock_output;" \
  -c "SELECT pg_reload_conf();"

The value must be unquoted here, and pg_reload_conf() is the whole procedure — do not restart a 1.x node to apply it. ALTER SYSTEM adds the quoting for you; quoting it yourself stores a single bogus plugin literally named "pgoutput, test_decoding, spock_output" and the error persists (confirm with SHOW output_plugin_libraries; — the output must contain no quotation marks). And the pre-2.0.0 startup routine drops every replication slot that is momentarily inactive, on every peer it can reach, so restarting even one 1.x node breaks the mesh in a way that does not heal on its own. Version 2.0.0 fixes that.

Subscriptions move to replicating shortly after the reload. Check on every node:

SELECT subscription_name, status FROM spock.sub_show_status();

Upgrading from 2.x

Version 3.0.0 replaces the pgcat pooler with PgBouncer and makes the HAProxy failover tier permanent. The client endpoint changes from RELEASE_NAME-pgcat to RELEASE_NAME-pgbouncer; the port stays 5432. The pgEdge nodes, their volumes and their data are kept.

Edit your values

Rename pgcat: to pgbouncer:, and delete image and routing from that block. Delete proxy.enabled. The chart refuses to render while a pgcat key, a pgcat image, routing or proxy.enabled: false remains, so an upgrade with stale values stops before anything changes:

pgedge 3.0.0: the pgcat pooler was REPLACED by PgBouncer and the `pgcat` values key was removed.

Optionally shorten the client gap

Point applications at RELEASE_NAME-pgedge-proxy.GVC_NAME.cpln.local:5432 first. That is HAProxy, present on any 2.2.0 install with the default proxy.enabled: true. It is unpooled, so keep the connection count well below what each node accepts.

Upgrade the release

cpln helm upgrade RELEASE_NAME oci://ghcr.io/controlplane-com/templates/pgedge \
  --version 3.0.0 --gvc GVC_NAME -f values.yaml

Every pgEdge node restarts once, one at a time within each location — with a single node per location, that is a write interruption of about a minute. The RELEASE_NAME-pgcat workload is replaced by RELEASE_NAME-pgbouncer, and a release that ran with proxy.enabled: false gains the HAProxy tier.

Point applications at the new endpoint

RELEASE_NAME-pgbouncer.GVC_NAME.cpln.local:5432.

Take a backup

Backups change format in 3.0.0 (one database, pg_dump custom format, .dump), and the restore job reads only the new format. Run a backup right after upgrading — see Backing Up.

pgcat.routing: single-writer has no equivalent: every location now writes to its own node, which was already the default from 2.1.0. A release still on 2.0.x routed every write to the first location’s first node, so if your application depends on a single write target, plan for multi-master writes before upgrading.

Scaling and Availability

What each failure needs. Your application never has to pick another node for anything inside a location: the template handles that, and the application needs only the ordinary retry described below. The one thing the template cannot handle for you is the loss of a whole location, because the pooled endpoint always reaches the PgBouncer in the client’s own location and PgBouncer has no per-location address.

Failure	Handled by	What it needs
A pgEdge node	The template (the HAProxy failover tier moves to another local node, or to a remote location if every local node is down)	3 nodes per location in production. New connections fail for about 3 seconds while HAProxy switches
A PgBouncer or HAProxy replica	The template	2 replicas of each per location (the default)
An upgrade or restart of the nodes	The template (nodes restart one at a time within a location; HAProxy carries traffic)	At least 2 nodes per location. New connections fail for a few seconds when a location’s `replica-0` restarts; with 1 node per location, writes stop for about a minute
A whole location	The template for the database — the other locations keep serving; your application for its own clients	At least 2 locations, and one of: your application running in at least 2 of them behind a public endpoint that spans them — a workload’s canonical endpoint already sends each request to the nearest healthy location; or, for an application in the location whose database tier is down, connecting directly to a node in another location at `replica-N.RELEASE_NAME-pgedge.LOCATION_NAME.GVC_NAME.cpln.local:5432`

A direct connection to a remote node is unpooled and counts against that node’s 300 connections, of which the poolers can use up to defaultPoolSize × maxReplicas × number of locations. Keep the application’s own pool small, and switch back to the pooled endpoint once its location recovers. Retry logic your application needs. Every row above can still fail a connection or a statement for a moment:

Retry failed connections with a short backoff (about 0.5–1 seconds) for 10–15 seconds. New connections fail for a few seconds while HAProxy moves off a node, and a PgBouncer replica that dies drops the connections going through it; new connections go to the other replica straight away.
Retry a failed statement, and make retried writes idempotent: a statement running on a process that fails returns an error, and may already have committed.

Spock replicates asynchronously, so writes a lost location had not yet replicated reach the others when it returns. Nodes per location. Use at least 3 nodes per location in production so the failover tier has a local node to fail over to; a single node per location fails over to a remote location instead. All nodes in all locations are writable at all times. Failback. When a failed node recovers, connections PgBouncer already holds keep using the failover target until they are recycled, so traffic returns to the local node within about 5 minutes rather than instantly. Upgrades and restarts. A change that restarts the node workload restarts the nodes one at a time within each location, while locations restart in parallel. With at least 2 nodes per location the failover tier carries traffic throughout; with a single node per location every location loses its node together, so plan a write interruption of about a minute. Data is preserved on each node’s volume, and the mesh reconciles itself on boot.

Self-repair restores replication, not history. When a broken subscription is rebuilt, replication resumes from the current position — rows written on a node while its subscription was down are not backfilled, and a node that took writes while cut off needs those rows reconciled by hand.

Growing or shrinking the cluster. Changing locations after data exists is not a verified path. The template creates subscriptions without initial synchronization, so a node added to a cluster that already holds data starts without its tables and rows and receives only changes made after it joins. Plan the final topology before loading data, or rebuild through a new release and a restore. If you remove a location from the GVC, remove it from locations in the same change; otherwise every node logs a WARNING about the missing location on each restart. Seeing which node serves a location. HAProxy’s stats page shows which node is active, which are backups, and which are UP or DOWN:

cpln port-forward RELEASE_NAME-pgedge-proxy 8405:8405 --gvc GVC_NAME

Then open http://localhost:8405/stats in a browser. Storage. Each node’s volume starts at volumeset.capacity GiB. Enable volumeset.autoscaling to grow volumes automatically up to maxCapacity; the volume set is never shrunk by the template.

Troubleshooting

The deployment never starts and cpln logs returns nothing

Symptom: no replica ever becomes ready, and cpln logs for the node, PgBouncer or HAProxy workload returns zero lines.Cause: the credentials secret named by postgres.credentialsSecretName does not exist, so the container is never started and has nothing to log.Fix: read the message, which names the missing secret:

cpln workload get-deployments RELEASE_NAME-pgedge --gvc GVC_NAME -o yaml

Create the secret as described in Prerequisites. The deployment recovers on its own within several minutes, or run cpln workload force-redeployment RELEASE_NAME-pgedge --gvc GVC_NAME.

Nodes restart with FATAL locations declared in values are not in GVC

Symptom: the install succeeds, but the node containers exit and restart, and the node logs show:

[pgedge] FATAL: locations declared in values are not in GVC 'GVC_NAME': aws-eu-central-1
[pgedge] GVC 'GVC_NAME' has: aws-us-west-2 aws-us-east-2

Cause: locations names a location the GVC does not have. The platform does not validate this at install, so the nodes check it themselves at startup and refuse to initialize a cluster whose declared members can never exist.Fix: either add the location to the GVC or remove it from locations, then upgrade the release. Read the lines with a server-side filter:

cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-pgedge"} |= "[pgedge]"' --limit 50 --since 10m

A node that has already initialized logs [pgedge] WARNING: locations declared in values are not in GVC instead and keeps serving, so this check never takes down a running cluster. A line reading [pgedge] WARNING: could not read the location list of GVC means the check could not run at all and was skipped — verify the GVC’s locations yourself with cpln gvc get GVC_NAME -o yaml.

helm upgrade fails with the pgcat pooler was REPLACED by PgBouncer

Symptom: an upgrade to 3.0.0 stops at render with pgedge 3.0.0: the pgcat pooler was REPLACED by PgBouncer, proxy.enabled was REMOVED, pgbouncer.routing does not exist or pgbouncer.image is a pgcat image.Cause: your values file still carries 2.x keys. Nothing was changed — the chart refuses before any resource is touched.Fix: edit the values as described in Upgrading from 2.x and run the upgrade again.

Connections fail with unsupported startup parameter in options

Symptom: a client that sets options (for example PGOPTIONS='-c some_setting=value') is refused at connect with unsupported startup parameter in options.Cause: PgBouncer passes only a fixed list of settings through options; see PgBouncer Connection Pooler.Fix: remove the setting from options and set it with ALTER ROLE … SET on every node, or with SET at the start of the session in session mode.

CREATE TABLE fails with cannot be added to replication set default

Symptom:

ERROR:  table events cannot be added to replication set default
DETAIL:  table does not have PRIMARY KEY and given replication set is configured to replicate UPDATEs and/or DELETEs

Cause: the template adds every new table to the default replication set, which replicates UPDATE and DELETE; Spock cannot do that for a table without a primary key.Fix: give the table a PRIMARY KEY — a uuid column with DEFAULT gen_random_uuid() is the recommended shape (see Primary Keys).

A table exists on one node only or rows written on one node never appear on the others

Symptom: a table you created is missing on other nodes, or the other nodes’ subscriptions stop at down after you created it.Cause: DDL does not replicate. A plain CREATE TABLE applied to one node only, or spock.replicate_ddl() was run without the follow-up spock.repset_add_table() on the other nodes.Fix: follow Schema Changes and DDL — create the table on every node directly, or broadcast it and then add it to the replication set on every other node. Confirm with SELECT * FROM spock.tables WHERE relname = 'TABLE_NAME'; on each node.

spock.sub_show_status() shows subscriptions down

Symptom: SELECT subscription_name, status FROM spock.sub_show_status(); returns down for one or more rows, and writes made on one node do not appear on another.Causes and fixes, in order of likelihood:

A node that is still booting or restarting — the mesh reconciles itself on boot. Give it a few minutes and re-check; rows written while a subscription was down are not backfilled (see Scaling and Availability).
A table created on one node only — see the entry above.
A 1.x data directory — the server log (not cpln logs) shows library "spock_output" may not be used as an output plugin. Apply the fix under Switching Replication On for an Existing 1.x Cluster.

A restore fails with already has objects or no backups of release

Symptom: cpln workload cron get shows the restore run failed, and the backup job’s log reads one of:

[pgedge-backup] FATAL: node replica-0.RELEASE_NAME-pgedge.LOCATION_NAME.GVC_NAME.cpln.local already has 10 object(s) (tables, views, sequences, functions, types or schemas) in 'mydb'. Restore only into an empty database: install a new release, or drop them on EVERY node first.
[pgedge-backup] FATAL: no backups of release 'RELEASE_NAME' (pgedge-RELEASE_NAME-*.dump) under aws://BUCKET/PREFIX/ -- set RESTORE_FILE to restore another release's backup

Cause: the restore runs only into a database that is empty on every node, and without RESTORE_FILE it picks only this release’s own backups. It refused before changing anything.Fix: drop your objects on every node as described in Restoring a Backup, or set RESTORE_FILE to the backup you want — for another release’s backup, the path inside the bucket.

PgBouncer exits with database name pgbouncer is reserved

Symptom: the RELEASE_NAME-pgbouncer replicas restart, and their log shows

[pgbouncer] FATAL: database name 'pgbouncer' is reserved for PgBouncer's admin console; use another name in your credentials secret

.Cause: the database entry of your credentials secret is pgbouncer.Fix: use a different database name. The nodes create the database on first boot, so on an install that has already initialized, reinstall under a new release with the corrected secret.

Important Notes

Your application must retry failed connections and statements — without it, every upgrade and replica restart shows up as errors in your app. Retry with a short backoff for 10–15 seconds and make retried writes idempotent; see Scaling and Availability.
3.0.0 changes the client endpoint to RELEASE_NAME-pgbouncer, and backups change format — take a backup right after upgrading. See Upgrading from 2.x.
Never run helm upgrade on a 1.x release against any later version — it deletes the GVC the 1.x chart created and everything in it. Migrate to a new release instead: Upgrading from 1.x.
The GVC must already contain every location you list, and may contain more. A missing location is not caught at install — the nodes exit with [pgedge] FATAL: locations declared in values are not in GVC .... Locations you do not list start nothing.
DDL does not replicate — create every table on every node (or broadcast it and add it to the replication set on the others), and give every table a PRIMARY KEY or CREATE TABLE fails outright.
Use at least 3 nodes per location in production, and at least 2 for upgrades without a write interruption.
Surviving a location loss needs your application in at least 2 locations, or a direct connection to a remote node — clients reach only their own location’s PgBouncer. See Scaling and Availability.
With internal_access.type: workload-list, list only your clients — the release’s own workloads are added for you.
Fix the topology before loading data — nodes added to a cluster that already holds data do not receive existing rows.
Rotating the database password is not a supported path in this version. The password is written into each node’s Spock node and subscription definitions when they are created, so changing the secret alone would break replication; plan a new release and a restore if you must rotate.
Release names must be unique per org — secrets are org-wide, so two releases with the same name collide even in different GVCs.
Conflict resolution is last-update-wins by commit timestamp. Route writes for a given entity to one node in your application if you need stronger guarantees.

External References

pgEdge Documentation

Official pgEdge Distributed PostgreSQL documentation

Spock Documentation

Spock multi-master replication extension, including the SQL function reference

PgBouncer Configuration

Pool modes, connection limits and startup parameters

PgBouncer Usage and Admin Console

Admin console commands such as SHOW POOLS and RECONNECT

pgEdge Template

View the source files, default values, and chart definition

​Overview

​What Gets Created

​Prerequisites

​Installation

UI

CLI

Terraform

Pulumi

​Configuration

​Locations and Replicas

​Image and Resources

​Credentials

​Volume Set

​HAProxy Failover Tier

​PgBouncer Connection Pooler

​Internal Access

​Backup

​Connecting

​Schema Changes and DDL

​Creating a Table

​Other DDL

​Primary Keys

​Operations

​Backing Up

​AWS S3 Prerequisites

​GCS Prerequisites

​Restoring a Backup

​Upgrading from 1.x

​Switching Replication On for an Existing 1.x Cluster

​Upgrading from 2.x

​Scaling and Availability

​Troubleshooting

​Important Notes

​External References

pgEdge Documentation

Spock Documentation

PgBouncer Configuration

PgBouncer Usage and Admin Console

pgEdge Template

Overview

What Gets Created

Prerequisites

Installation

Configuration

Locations and Replicas

Image and Resources

Credentials

Volume Set

HAProxy Failover Tier

PgBouncer Connection Pooler

Internal Access

Backup

Connecting

Schema Changes and DDL

Creating a Table

Other DDL

Primary Keys

Operations

Backing Up

AWS S3 Prerequisites

GCS Prerequisites

Restoring a Backup

Upgrading from 1.x

Switching Replication On for an Existing 1.x Cluster

Upgrading from 2.x

Scaling and Availability

Troubleshooting

Important Notes

External References