> ## Documentation Index
> Fetch the complete documentation index at: https://docs.controlplane.com/llms.txt
> Use this file to discover all available pages before exploring further.

# pgEdge

> pgEdge active-active PostgreSQL 17 with Spock multi-master replication across the locations you list, with a per-location HAProxy failover tier and PgBouncer pooler. Deploys into an existing GVC that already has those locations; a dictionary credentials secret must exist first.

## Overview

pgEdge is an active-active distributed PostgreSQL cluster built on the Spock multi-master replication extension. Every node accepts reads and writes, and a row written on any node replicates to all the others with last-update-wins conflict resolution. This template deploys one set of PostgreSQL 17 nodes in each location you list, a per-location HAProxy failover tier that keeps a location serving when one of its nodes fails, a per-location PgBouncer connection pooler that gives applications a single endpoint, and an optional scheduled backup to S3 or GCS that also restores with one command.

Database credentials are **not** template values. The nodes, the pooler and the backup job read the username, password and database name from a [dictionary secret](/guides/create-secret/dictionary) you create before installing, so no password passes through Helm or lands in the release.

<Note>
  This template deploys into an existing [GVC](/reference/gvc) that you already have; it does not create, provision or manage one. Every location you list in `locations` must already be one of that GVC's locations. A GVC location you do not list starts nothing — its deployment reads `This workload location is deactivated because maxScale is set to 0.`, which is what a healthy install looks like in a shared GVC, not a fault.
</Note>

### What Gets Created

| Resource | Name | Purpose | Optional |
| - | - | - | - |
| Workload | `RELEASE_NAME-pgedge` | The PostgreSQL 17 + Spock nodes (stateful, `locations[].replicas` nodes in each listed location, TCP `5432`). Every node is writable and individually addressable | — |
| Workload | `RELEASE_NAME-pgedge-proxy` | Per-location HAProxy failover tier (standard, `proxy.minReplicas` to `proxy.maxReplicas` per location, TCP `5432`, plus HTTP `8404` health and `8405` stats). Each location's PgBouncer connects to it | — |
| Workload | `RELEASE_NAME-pgbouncer` | PgBouncer connection pooler, the endpoint applications connect to (standard, `pgbouncer.minReplicas` to `pgbouncer.maxReplicas` per location, autoscales on requests per second, TCP `5432`) | — |
| Workload | `RELEASE_NAME-pgedge-backup` | Scheduled `pg_dump` of your database to S3 or GCS, and the one-command restore (cron). Runs only in the first entry of `locations` | `backup.enabled: true` |
| Volume Set | `RELEASE_NAME-pgedge-vs` | One `ext4` general-purpose SSD volume per node. Snapshotted on `volumeset.snapshots.schedule` (daily at 03:00 UTC by default), kept for `volumeset.snapshots.retentionDuration`, with a final snapshot on deletion | — |
| Identity | `RELEASE_NAME-pgedge-identity` | Shared by every workload in the release; carries the cloud account binding when backups are enabled | — |
| Policy | `RELEASE_NAME-pgedge-policy` | Grants the identity `reveal` on this release's secrets and on your credentials secret, and on nothing else | — |
| Policy | `RELEASE_NAME-pgedge-gvc-policy` | Grants the identity `view` on exactly the one GVC the release is installed into, so each node can confirm at startup that the GVC has every location in `locations` | — |
| Secret | `RELEASE_NAME-pgedge-startup` | The node startup script (initialization, Spock node and subscription setup) | — |
| Secret | `RELEASE_NAME-pgedge-config` | The backup destination (bucket and region) | `backup.enabled: true` |

Your credentials secret is created by you and is never part of the release.

## Prerequisites

Two things must exist before you install: a GVC that already has the locations you intend to use, and the credentials secret. Secrets are org-level, so no GVC flag is involved when creating one.

<Steps>
  <Step title="A GVC that contains every location you will list">
    The template deploys into the GVC you pass to `--gvc`. It may have more locations than you list — nothing pgEdge-related runs in those — but it must have every one you do list. Check its locations under `spec.staticPlacement.locationLinks`:

    ```bash theme={null}
    cpln gvc get GVC_NAME -o yaml
    ```

    Set `locations` to a subset of what you see there. A location that is not on the GVC is not caught at install time — the install succeeds and the nodes then refuse to initialize (see [Troubleshooting](#troubleshooting)).
  </Step>

  <Step title="Create the database credentials secret">
    A [dictionary secret](/guides/create-secret/dictionary) holding exactly three keys — `username`, `password` and `database`. The nodes create that user and that database on first boot, PgBouncer serves only that database, and its admin console accepts the same username and password. Do not name the database `pgbouncer`; that name is reserved for the admin console.

    ```bash theme={null}
    cpln secret create-dictionary --name SECRET_NAME \
      --entry username=myuser \
      --entry password='YOUR-STRONG-PASSWORD' \
      --entry database=mydb
    ```

    Set `postgres.credentialsSecretName` to the name you used. Secret names are org-wide, so give each release its own.
  </Step>

  <Step title="Read the secret back later">
    Pass `-o yaml`. A bare `cpln secret reveal` prints only a summary table, not the values:

    ```bash theme={null}
    cpln secret reveal SECRET_NAME -o yaml
    ```
  </Step>
</Steps>

<Warning>
  **Create the secret before installing, or the deployment wedges silently.** The template refuses to render when `postgres.credentialsSecretName` is blank, but a name that points at a secret which does not exist installs "successfully" and then never starts. The container never runs, so `cpln logs` returns **zero lines** — there is nothing to log. The one place the reason appears is `status.versions[].message`:

  ```bash theme={null}
  cpln workload get-deployments RELEASE_NAME-pgedge --gvc GVC_NAME -o yaml
  ```

  Use `get-deployments` — plain `cpln workload get` has no `versions` key. Creating the missing secret repairs the deployment on its own within several minutes, or run `cpln workload force-redeployment RELEASE_NAME-pgedge --gvc GVC_NAME` to skip the wait.
</Warning>

Backups need a bucket and, for AWS or GCP, a Control Plane Cloud Account before they can be enabled — see [Backing Up](#backing-up). Nothing else is required.

## Installation

The default `locations` list names three AWS regions with three nodes each, so a real install nearly always carries its own location list. Lists cannot be set piecemeal with `--set` — Helm replaces the whole list, and a `--set locations[0].name=...` without `replicas` fails to render with `location 'aws-us-east-1' must have at least 1 replica` — so put them in a values file:

```yaml theme={null}
# values.yaml
locations:
  - name: aws-us-east-1
    replicas: 1
postgres:
  credentialsSecretName: SECRET_NAME
```

```bash theme={null}
cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/pgedge \
  --version 3.0.0 --gvc GVC_NAME \
  -f values.yaml
```

To install, follow the instructions for your preferred method:

<CardGroup cols={2}>
  <Card title="UI" href="/template-catalog/install-manage/ui" icon="laptop">
    Browse, install, and manage templates visually
  </Card>

  <Card title="CLI" href="/template-catalog/install-manage/cli" icon="terminal">
    Manage templates from your terminal
  </Card>

  <Card title="Terraform" href="/template-catalog/install-manage/terraform" icon={<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 128 128"><g fill-rule="evenodd"><path d="M77.941 44.5v36.836L46.324 62.918V26.082zm0 0" fill="#5c4ee5"/><path d="M81.41 81.336l31.633-18.418V26.082L81.41 44.5zm0 0" fill="#4040b2"/><path d="M11.242 42.36L42.86 60.776V23.941L11.242 5.523zm0 0M77.941 85.375L46.324 66.957v36.82l31.617 18.418zm0 0" fill="#5c4ee5"/></g></svg>}>
    Declare templates in your Terraform configurations
  </Card>

  <Card
    title="Pulumi"
    href="/template-catalog/install-manage/pulumi"
    icon={<svg xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 24 24" id="Pulumi-Icon--Streamline-Svg-Logos" height="24" width="24">
    <desc>
        Pulumi Icon Streamline Icon: https://streamlinehq.com
    </desc>
    <path fill="#f26e7e" d="M4.683025 13.3318c0.869125 -0.5018 0.870575 -2.1264 0.003225 -3.62865s-2.27504 -2.313275 -3.1441725 -1.811475C0.672945 8.3935 0.6715 10.0181 1.53885 11.52035c0.86735 1.502275 2.27505 2.313275 3.144175 1.81145Zm0.0052 3.2167c0.86735 1.502275 0.865925 3.126875 -0.003225 3.628675 -0.86915 0.5018 -2.2768275 -0.309225 -3.144175 -1.81145 -0.8673525 -1.50225 -0.8659075 -3.126875 0.003225 -3.628675 0.8691325 -0.5018 2.276825 0.309225 3.144175 1.81145Zm5.922875 3.4243c0.86735 1.50225 0.8659 3.126775 -0.003225 3.62875 -0.869125 0.501775 -2.27685 -0.309325 -3.1442 -1.81155 -0.867325 -1.50225 -0.865875 -3.12685 0.00325 -3.628675 0.869125 -0.5018 2.276825 0.309225 3.144175 1.811475Zm-0.001925 -6.845275c0.86735 1.50225 0.8659 3.12685 -0.003225 3.628675 -0.869125 0.5018 -2.276825 -0.309225 -3.144175 -1.811475 -0.86735 -1.50225 -0.8659 -3.12685 0.003225 -3.62865 0.869125 -0.501825 2.276825 0.3092 3.144175 1.81145Z" stroke-width="0.25"></path>
    <path fill="#8a3391" d="M22.45775 11.524125c0.86725 -1.502225 0.865925 -3.12685 -0.003225 -3.62865 -0.869125 -0.501825 -2.276825 0.3092 -3.144175 1.811475 -0.86735 1.50225 -0.8659 3.126825 0.003225 3.62865 0.869125 0.501825 2.276825 -0.3092 3.144175 -1.811475Zm0.000175 3.2151c0.869075 0.5018 0.870625 2.1264 0.003225 3.62865 -0.86735 1.50225 -2.27505 2.313275 -3.144175 1.81145 -0.869125 -0.5018 -0.870575 -2.126425 -0.003225 -3.62865 0.86735 -1.50225 2.27505 -2.313275 3.144175 -1.81145ZM16.536225 18.157875c0.86915 0.501825 0.8706 2.126425 0.00325 3.628675 -0.86735 1.502125 -2.275075 2.313225 -3.1442 1.81145 -0.869125 -0.50175 -0.870575 -2.126425 -0.003225 -3.62865 0.867375 -1.502275 2.27505 -2.3133 3.144175 -1.811475Zm-0.003325 -6.843775c0.869125 0.5018 0.870575 2.126425 0.003225 3.628675s-2.27505 2.313275 -3.1442 1.811475c-0.869125 -0.501825 -0.870575 -2.126425 -0.003225 -3.628675 0.86735 -1.502275 2.27505 -2.313275 3.1442 -1.811475Z" stroke-width="0.25"></path>
    <path fill="#f7bf2a" d="M15.138225 2.06721c0 1.003615 -1.40625 1.817215 -3.14095 1.817215 -1.7347 0 -3.14095 -0.8136 -3.14095 -1.817215C8.856325 1.06359 10.262575 0.25 11.997275 0.25c1.7347 0 3.14095 0.81359 3.14095 1.81721ZM9.2166 5.482375c0 1.003625 -1.40625 1.8172 -3.14095 1.8172 -1.7347 0 -3.14095 -0.813575 -3.14095 -1.8172s1.40625 -1.817225 3.14095 -1.817225c1.7347 0 3.14095 0.8136 3.14095 1.817225Zm8.71005 1.8172c1.7347 0 3.14095 -0.813575 3.14095 -1.8172s-1.40625 -1.817225 -3.14095 -1.817225c-1.7347 0 -3.14095 0.8136 -3.14095 1.817225s1.40625 1.8172 3.14095 1.8172Zm-2.788425 1.605625c0 1.003625 -1.40625 1.8172 -3.14095 1.8172 -1.7347 0 -3.14095 -0.813575 -3.14095 -1.8172 0 -1.0036 1.40625 -1.8172 3.14095 -1.8172 1.7347 0 3.14095 0.8136 3.14095 1.8172Z" stroke-width="0.25"></path>
    </svg>}
  >
    Declare templates in your Pulumi programs
  </Card>
</CardGroup>

## Configuration

### Locations and Replicas

Each entry pairs a location with a node count. Every location listed must already be on the GVC you install into; extra GVC locations run nothing. Listing the same location twice is rejected at render, because duplicates would produce duplicate Spock node names.

```yaml theme={null}
locations: # For replicas: use 1 for dev/testing, 3 for production
  - name: aws-us-west-2
    replicas: 3
  - name: aws-us-east-2
    replicas: 3
  - name: aws-eu-central-1
    replicas: 3

multiZone: false  # Set to true to spread replicas across availability zones within each location
```

| Environment | Replicas per location |
| - | - |
| Dev / testing | 1 |
| Production | 3 |

The **first** entry of `locations` is special: it is the only location the backup job runs in.

Verify that your locations support multiple availability zones before enabling `multiZone`.

### Image and Resources

The node container exposes both a reservation and a limit. `maxCpu: 2` against `minCpu: 500m` is exactly the 4:1 ratio a stateful workload allows, so raise `minCpu` whenever you raise `maxCpu`.

```yaml theme={null}
image: ghcr.io/pgedge/pgedge-postgres:17-spock5-standard

resources:
  minCpu: 500m
  minMemory: 1Gi
  maxCpu: 2
  maxMemory: 4Gi
```

Each node accepts 300 connections. The template sets this; it is not a values key.

### Credentials

```yaml theme={null}
postgres:
  credentialsSecretName: my-pgedge-credentials
```

The named [dictionary secret](/guides/create-secret/dictionary) must exist before install and hold `username`, `password` and `database` — see [Prerequisites](#prerequisites). The nodes, PgBouncer, HAProxy's health checks and the backup job all read it; nothing else in the release carries a credential.

### Volume Set

Set the initial storage capacity per node (minimum 10 GiB), the volume snapshot schedule, and whether volumes grow as data grows:

```yaml theme={null}
volumeset:
  capacity: 10  # Initial capacity in GiB (minimum is 10)
  snapshots:
    schedule: "0 3 * * *"    # volume snapshots of every node, daily at 3am UTC; at most hourly; "" turns them off
    retentionDuration: 7d    # how long each snapshot is kept
  autoscaling:
    enabled: false  # Set to true to enable autoscaling
    maxCapacity: 100  # Maximum capacity in GiB when autoscaling is enabled
    minFreePercentage: 10  # Minimum free percentage to trigger scaling
    scalingFactor: 1.2  # How much to scale up when triggered
```

`snapshots.schedule` must be a cron expression with a single fixed minute; anything more frequent than hourly is refused at render. A final snapshot of each volume is taken when the volume set is deleted, whether or not a schedule is set.

### HAProxy Failover Tier

The request flow is `app → PgBouncer (pooler) → HAProxy (failover) → pgEdge node`. PgBouncer pools connections; HAProxy picks the node. Both run one set per location, and the endpoint applications connect to is the PgBouncer one.

Each location's HAProxy sends all of that location's traffic to **its own** first node (`replica-0`) and, if that node is unhealthy, fails over in order to the other local nodes, then to the nodes of the other locations in `locations` order — so applications keep serving through a node failure without changing anything. Every location writes to its own node, so the cluster is active/active. The tier is always on.

```yaml theme={null}
proxy:
  image: haproxy:3.0.28  # pinned exact; Debian variant (perl needed by the startup check)
  resources:
    cpu: 100m
    memory: 128Mi
  minReplicas: 2         # per location
  maxReplicas: 2         # per location
```

All of a location's statements land on whichever node HAProxy currently prefers, so local reads are not spread across a location's other nodes; that is the deliberate trade for read-your-writes consistency within a location.

### PgBouncer Connection Pooler

PgBouncer multiplexes application connections into a smaller pool of real database connections, protecting PostgreSQL from connection exhaustion under high concurrency. It serves only the database named in your credentials secret. `minReplicas` and `maxReplicas` are per location, and PgBouncer autoscales between them on request rate.

```yaml theme={null}
pgbouncer:
  image: ghcr.io/cloudnative-pg/pgbouncer:1.26.0-202610011121-trixie  # pinned immutable build
  poolMode: transaction  # options: session, transaction, statement
  defaultPoolSize: 20    # real Postgres connections per PgBouncer replica; node max_connections is 300
  maxClientConn: 1000    # client connections accepted per PgBouncer replica
  resources:
    cpu: 500m            # PgBouncer is single-threaded: add replicas rather than cores
    memory: 128Mi
  minReplicas: 2         # per location
  maxReplicas: 4         # per location
```

**Connection budget.** Every PgBouncer replica in a location pools onto that location's `replica-0`, and each node accepts 300 connections. The defaults use at most 20 × 4 = 80 per location, so one node can carry three locations' pools after a failover. If you raise `defaultPoolSize` or `maxReplicas`, keep `defaultPoolSize × maxReplicas × number of locations` under about 290.

| Pool mode | Behavior |
| - | - |
| `transaction` | Connection held only for the duration of a transaction. Best for most web and API workloads. Protocol-level prepared statements work; session-level `SET`, advisory locks and `LISTEN` do not. A temporary table works only within a single transaction (`CREATE TEMP TABLE … ON COMMIT DROP`): one created outside a transaction stays on the pooled server connection and is visible to other clients. |
| `session` | Connection held for the entire client session. Compatible with all PostgreSQL features but provides less connection reuse. A session idle for more than 1 hour is closed by the failover tier; clients reconnect. |
| `statement` | Connection returned after every statement. Multi-statement transactions are rejected. Rarely used. |

**Session settings in the connection string.** In `transaction` mode (the default), clients may pass these through the `options` startup parameter (`PGOPTIONS='-c statement_timeout=5s'`, libpq `options=`, JDBC `options=`), and PgBouncer applies them on every server connection the client uses: `statement_timeout`, `lock_timeout`, `idle_in_transaction_session_timeout`, `idle_session_timeout`, `work_mem`, `maintenance_work_mem`, `default_transaction_isolation`, `client_min_messages`, plus `search_path`, `application_name`, `TimeZone`, `DateStyle` and `client_encoding`. Any other setting in `options` is refused at connect with `unsupported startup parameter in options` — set it with `ALTER ROLE … SET` instead.

In `session` mode, `options` is reliably applied only on a brand-new server connection; a client handed a pooled connection silently gets the server defaults. Run `SET` at the start of each session instead, or `ALTER ROLE … SET` on **every** node — that reaches new server connections only, so run `RECONNECT` on the PgBouncer admin console or wait up to 5 minutes.

### Internal Access

Controls which workloads can connect to the nodes, the failover tier and PgBouncer. Nothing in this template is exposed publicly, and there is no public access option.

```yaml theme={null}
internal_access: # Applies to the pgedge, PgBouncer and HAProxy workloads
  type: same-gvc  # options: same-gvc, same-org, workload-list
  workloads:  # For workload-list: list only your clients --
              # this release's own workloads are added automatically
    #- //gvc/GVC_NAME/workload/WORKLOAD_NAME
```

* `same-gvc` — Allow access from all workloads in the same GVC.
* `same-org` — Allow access from all workloads in the org.
* `workload-list` — Allow access only from the workloads you list, **plus this release's own workloads**, which the template adds for you.

<Warning>
  **With `workload-list`, list only your own client workloads.** The same list governs Spock replication between the nodes, HAProxy's and PgBouncer's connections, and the backup job's connection, so the template always adds this release's pgedge, PgBouncer, HAProxy and (when enabled) backup workloads to it. Anything of the release's you add by hand is de-duplicated.
</Warning>

### Backup

When `backup.enabled` is `true`, a cron workload dumps your database on the configured schedule and uploads it to S3 or GCS. The same workload performs restores. Keep `schedule` quoted; any valid cron expression works, including one beginning with `*`. The bucket, cloud account and IAM setup are described under [Backing Up](#backing-up).

```yaml theme={null}
backup:
  enabled: false
  image: ghcr.io/controlplane-com/backup-images/postgres-backup:17.1.0
  schedule: "0 2 * * *"   # daily at 2am UTC — runs in locations[0] only
  activeDeadlineSeconds: 21600  # a backup or restore still running after 6 h is stopped

  resources:
    cpu: 100m
    memory: 256Mi  # 128Mi measured OOM-killing the GCS upload

  provider: aws  # Options: aws or gcp

  aws:
    bucket: my-backup-bucket
    region: us-east-1
    cloudAccountName: my-backup-cloudaccount
    policyName: my-backup-policy
    prefix: pgedge/backups  # folder where backups will be stored

  gcp:
    bucket: my-backup-bucket
    cloudAccountName: my-backup-cloudaccount
    prefix: pgedge/backups  # folder where backups will be stored
```

## Connecting

<Warning>
  **Your application must retry — build it in from the start.** Restarts, upgrades and failovers are routine, and each one fails a few connections or statements for a few seconds even though the database stays up. Retry a failed connection with a short backoff (about 0.5–1 seconds) for 10–15 seconds, and retry a failed statement, making retried writes idempotent. Connection pools usually replace a broken connection but do not re-run the failed query for you. Details: [Scaling and Availability](#scaling-and-availability).
</Warning>

Connect through PgBouncer for all application traffic. The per-location HAProxy behind it handles node failover, and the endpoint does not change. Always use the fully-qualified `.GVC_NAME.cpln.local` form — the bare workload name does not resolve reliably from every workload type.

| What | Address | Notes |
| - | - | - |
| Pooled endpoint (use this) | `RELEASE_NAME-pgbouncer.GVC_NAME.cpln.local:5432` | Internal only. Each client reaches the PgBouncer in its own location |
| Unpooled, with failover | `RELEASE_NAME-pgedge-proxy.GVC_NAME.cpln.local:5432` | HAProxy directly. Counts against the node's 300 connections |
| A single node, directly | `replica-N.RELEASE_NAME-pgedge.LOCATION_NAME.GVC_NAME.cpln.local:5432` | Needed for schema changes and per-node checks. `N` runs from `0` to `replicas - 1`. Unpooled |
| PgBouncer admin console | Pooled endpoint, database `pgbouncer` | Same username and password as your credentials secret |
| Database, username, password | `cpln secret reveal SECRET_NAME -o yaml` | The `database`, `username` and `password` entries of your credentials secret. The pooler serves only that database |
| Public endpoint | None | The template has no public access option |

PgBouncer does not offer TLS — use `sslmode=disable` or `sslmode=prefer`; traffic stays inside the GVC. Clients must support SCRAM-SHA-256 authentication (libpq 10+, JDBC 42.2+).

To verify from your own machine, tunnel to PgBouncer and run a query against the tunnel:

```bash theme={null}
cpln port-forward RELEASE_NAME-pgbouncer 5432:5432 --gvc GVC_NAME
```

```bash theme={null}
psql "postgresql://USERNAME:PASSWORD@127.0.0.1:5432/DATABASE" -c 'select version()'
```

To check replication from inside a node, run `psql` on it — the container already carries the credentials in its environment:

```bash theme={null}
cpln workload exec RELEASE_NAME-pgedge --gvc GVC_NAME --replica RELEASE_NAME-pgedge-0 --location LOCATION_NAME --container pgedge -- bash -c 'psql -U "$PG_USERNAME" -d "$PG_DATABASE" -c "SELECT subscription_name, status FROM spock.sub_show_status();"'
```

On a healthy cluster of N nodes every node returns N−1 rows and each reads `replicating`.

### Schema Changes and DDL

Spock replicates row-level changes (`INSERT`, `UPDATE`, `DELETE`) automatically. **DDL does not replicate.** A plain `CREATE TABLE` or `ALTER TABLE` applies only to the node you ran it on; the other nodes never learn about it, and rows written into that table on one node cannot be applied on a node where it does not exist. Run DDL against a node directly, never through PgBouncer.

<Warning>
  **Every table must have a `PRIMARY KEY`.** The template adds each new table to the `default` replication set automatically, and that set replicates `UPDATE` and `DELETE`, which Spock cannot do without a key. A table without one does not merely fail to replicate — the `CREATE TABLE` itself is rejected. Temporary and `UNLOGGED` tables are exempt: they are never replicated and stay on the node that created them.

  ```text theme={null}
  ERROR:  table events cannot be added to replication set default
  DETAIL:  table does not have PRIMARY KEY and given replication set is configured to replicate UPDATEs and/or DELETEs
  ```
</Warning>

#### Creating a Table

The simplest correct procedure is to run the same `CREATE TABLE` on **every** node. The auto-add trigger fires locally on each one, so the table ends up in the `default` replication set everywhere and DML replicates in all directions:

```sql theme={null}
-- Run on EVERY node, connecting to each directly (not through PgBouncer)
CREATE TABLE orders (
  id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  amount numeric,
  created_at timestamptz DEFAULT now()
);
```

For larger clusters you can broadcast the DDL instead — but it takes **two** steps, and the second one runs on the other nodes, not on the node that broadcast:

```sql theme={null}
-- Step 1: on ONE node -- creates the table on all nodes
SELECT spock.replicate_ddl('CREATE TABLE orders (
  id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  amount numeric,
  created_at timestamptz DEFAULT now()
);');

-- Step 2: on every OTHER node -- adds the table to that node's replication set
SELECT spock.repset_add_table('default', 'orders'::regclass);
```

Step 2 is needed because Spock suppresses event triggers while applying replicated changes, so the auto-add trigger fires only on the node that called `replicate_ddl`. Running step 2 on *that* node fails with `duplicate key value violates unique constraint "replication_set_table_pkey"`, and skipping it strands writes made on the other nodes. Confirm on every node — each must return a row:

```sql theme={null}
SELECT * FROM spock.tables WHERE relname = 'orders';
```

#### Other DDL

`ALTER TABLE` and `DROP TABLE` follow the same rule — apply on every node, or broadcast once:

```sql theme={null}
SELECT spock.replicate_ddl('ALTER TABLE orders ADD COLUMN status text DEFAULT ''pending'';');
SELECT spock.replicate_ddl('DROP TABLE orders;');
```

#### Primary Keys

Use `uuid` primary keys instead of `serial` or `bigserial`. Each node maintains its own sequence, so auto-increment integers collide when the same value is generated on two nodes at once:

```sql theme={null}
-- Good: globally unique, no conflicts
id uuid PRIMARY KEY DEFAULT gen_random_uuid()

-- Avoid: causes duplicate key conflicts under concurrent multi-node writes
id serial PRIMARY KEY
```

## Operations

### Backing Up

When `backup.enabled` is `true`, the `RELEASE_NAME-pgedge-backup` cron workload dumps **your database** (the `database` entry of your credentials secret) with `pg_dump` in custom format and uploads it as `PREFIX/pgedge-RELEASE_NAME-YYYY-MM-DDTHH-MM-SSZ.dump`. Every node holds a full copy, so the job dumps the first node that accepts a login and moves on to the next if a dump fails. A run that fails part-way leaves no backup behind. The job runs only in the first location in `locations`, so backups pause while that whole location is down.

Roles and large objects (`lo_*`) are not included: Spock does not replicate large objects, and the user from your credentials secret owns everything that is restored.

To take a backup now, start the job in your first location (the first entry of `locations`):

```bash theme={null}
cpln workload cron start RELEASE_NAME-pgedge-backup --gvc GVC_NAME --location LOCATION_NAME
```

#### AWS S3 Prerequisites

1. Create your S3 bucket and note its name and region. Set `backup.aws.bucket` and `backup.aws.region`.
2. If you do not have a Cloud Account set up, refer to the docs to [Create a Cloud Account](/guides/create-cloud-account). Note the cloud account name.
3. Create a new AWS IAM policy with the following JSON (replace `YOUR_BUCKET_NAME`):

```json theme={null}
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:ListBucket",
                "s3:GetObjectVersion",
                "s3:DeleteObjectVersion",
                "s3:GetBucketLocation",
                "s3:AbortMultipartUpload",
                "s3:ListBucketMultipartUploads",
                "s3:ListMultipartUploadParts"
            ],
            "Resource": [
                "arn:aws:s3:::YOUR_BUCKET_NAME",
                "arn:aws:s3:::YOUR_BUCKET_NAME/*"
            ]
        }
    ]
}
```

4. Set `backup.aws.cloudAccountName` to your cloud account name.
5. Set `backup.aws.policyName` to the policy created in step 3.

<Note>
  **If your IAM policy predates template version 1.1.1**, update it to the full action list above. Earlier versions attached the AWS managed policy `ReadOnlyAccess` to the backup identity as well, which granted read access to every bucket in your account and silently supplied any read action your own policy omitted. The identity now carries `cpln-connector` and your bucket-scoped policy only.
</Note>

#### GCS Prerequisites

1. Create your GCS bucket and note its name. Set `backup.gcp.bucket`.
2. If you do not have a Cloud Account set up, refer to the docs to [Create a Cloud Account](/guides/create-cloud-account). Note the cloud account name.
3. Grant the cloud account's GCP service account the `Storage Admin` role. Control Plane uses it to give the backup job `roles/storage.objectAdmin` on this one bucket.
4. Set `backup.gcp.cloudAccountName` to your cloud account name.

### Restoring a Backup

The backup job also restores. It puts the schema on **every** node (DDL does not replicate), adds each table to the replication set, loads the data **once** so Spock replicates it, sets sequences on every node, waits until every node holds the same row counts, and finally refreshes materialized views on every node. It refuses to run unless every node is reachable and the database is **empty on every node**, so it can never merge into or overwrite data. Keep applications from writing to the database while it runs.

Restore this release's newest backup:

```bash theme={null}
cpln workload cron start RELEASE_NAME-pgedge-backup --gvc GVC_NAME --location LOCATION_NAME \
  --env PGEDGE_ACTION=restore
```

Or a specific one — a file name under the prefix, or a full path inside the bucket:

```bash theme={null}
cpln workload cron start RELEASE_NAME-pgedge-backup --gvc GVC_NAME --location LOCATION_NAME \
  --env PGEDGE_ACTION=restore --env RESTORE_FILE=pgedge-RELEASE_NAME-YYYY-MM-DDTHH-MM-SSZ.dump
```

Follow the run (status `successful` or `failed`):

```bash theme={null}
cpln workload cron get RELEASE_NAME-pgedge-backup --gvc GVC_NAME
```

Read the log for the reason a run failed:

```bash theme={null}
cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-pgedge-backup", container="backup-pgedge"} |= "pgedge-backup"' --since 1h --limit 200
```

* **Recovering into a new release** (lost cluster, new GVC): install with `backup.enabled: true` against the same bucket, then restore with `RESTORE_FILE=OLD_PREFIX/pgedge-OLD_RELEASE-….dump`.
* **Restoring over an existing database, or retrying a restore that failed part-way**: drop your objects (tables, views, sequences, functions, types, non-public schemas) on **every** node first, connecting to each node directly. Tables in a replication set need `CASCADE` (`DROP TABLE orders CASCADE;`). Do not drop the `public` schema: it holds the template's auto-replication trigger, which a node recreates only when it is first created.
* The archive is downloaded to the job's local disk before it is applied. For a large database, raise `backup.activeDeadlineSeconds`.
* Ownership and `GRANT`s are not restored; every object belongs to the credentials user.
* Only backups written by template version 3.0.0 or later can be restored this way. The `.sql.gz` files earlier versions wrote are not read by the job.

Each node's volume is also snapshotted on `volumeset.snapshots.schedule`. Restoring a single node's volume into a running multi-master cluster has not been verified and is not a substitute for the restore above.

### Upgrading from 1.x

Template versions through 1.1.1 created their own GVC, so that GVC is part of a 1.x release's manifest. Version 2.0.0 and later do not declare one — and Helm deletes what a chart stops declaring.

<Warning>
  **An in-place `helm upgrade` from 1.x to any later version is not supported and destroys the cluster.** The upgrade would delete the GVC the 1.x chart created together with every workload, volume set and identity inside it — including your data. The only supported path is the migration below: install the current version as a new release into an existing GVC, move the data, then remove the old release.

  The chart refuses to render when your values still carry the 1.x `gvc` key, so an upgrade that passes your old values file stops before a single API call is made:

  ```text theme={null}
  Error: execution error at (pgedge/templates/identity.yaml:2:4): pgedge 2.0.0: the `gvc` values key was REMOVED. This chart no longer creates a GVC -- it deploys into the GVC you install into, and `gvc.locations` moved to the top-level `locations`. DO NOT `helm upgrade` a 1.x release onto 2.0.0: the upgrade drops `kind: gvc` from the manifest and Helm deletes what a chart no longer declares, which DESTROYS that GVC and every workload, volumeset and identity inside it. Install 2.0.0 as a NEW release against an existing GVC, move your data, then uninstall the old release. See `Migrating from 1.x` in the README.
  ```

  **The guard cannot cover an upgrade run with no values at all.** A 1.x release installed on pure defaults has no `gvc` key for the chart to see, so nothing fires. Do not run `helm upgrade` on a 1.x release against any later chart under any circumstances.
</Warning>

The 1.0.x credential values (`postgres.username`, `postgres.password`, `postgres.database`) are refused at render as well, with their own message, so pasting an old values file into a new install produces an error rather than a silent misconfiguration.

<Steps>
  <Step title="Back up the old cluster">
    Use `backup.enabled` on the 1.x release, or take a manual `pg_dump` against one of its nodes directly, at `replica-0.OLD_RELEASE-pgedge.LOCATION_NAME.OLD_GVC.cpln.local`.

    If replication on the old cluster was never switched on — which is every 1.x install that did not apply the fix below — the nodes may hold **different** data. Compare row counts per node before deciding which one to dump, or reconcile them first.
  </Step>

  <Step title="Choose the GVC for the new release">
    Pick a GVC that already has every location you intend to list in `locations`. See [Prerequisites](#prerequisites).
  </Step>

  <Step title="Create the credentials secret and install as a new release">
    Use a **different release name** — secret names are org-wide, so a same-named release collides with the 1.x one even in another GVC. Coming from 1.0.2 or earlier, the credentials were plain values with a published default password; choose a new password now and put it in the dictionary secret, since a fresh install creates its roles from that secret.
  </Step>

  <Step title="Load the data and cut over">
    A 1.x backup is not in the format the restore job reads, so load it into the new cluster by hand, respecting the rules in [Schema Changes and DDL](#schema-changes-and-ddl): the schema on every node first, then the data on **one** node only. This manual load is not covered by the restore job, so rehearse it on a scratch release first. Then point your applications at the new pooled endpoint, `NEW_RELEASE-pgbouncer.NEW_GVC.cpln.local:5432`.
  </Step>

  <Step title="Uninstall the old release against the GVC you installed it into">
    Not the GVC it created — the GVC you passed at install time is where Helm tracks the release, and uninstalling from there removes the created GVC with it.
  </Step>
</Steps>

#### Switching Replication On for an Existing 1.x Cluster

PostgreSQL 17.11 ships `output_plugin_libraries = 'pgoutput, test_decoding'`, and Spock's output plugin is not on that list. On every template version through 1.1.1 this means **no replication slot could ever be created**: each subscription sits at `down`, each node accepts writes that never leave it, and every status surface still reports healthy. The reason appears in the node's own PostgreSQL server log, not in `cpln logs`:

```text theme={null}
ERROR:  library "spock_output" may not be used as an output plugin
```

Version 2.0.0 and later write the setting into `postgresql.conf` during `initdb`, so a **fresh** install needs nothing. An existing data directory keeps the old setting, so if you are staying on 1.x while planning the migration, apply it by hand **on every node**, connecting to each node directly:

```bash theme={null}
psql "host=replica-0.RELEASE_NAME-pgedge.LOCATION_NAME.GVC_NAME.cpln.local user=USERNAME dbname=DATABASE" \
  -c "ALTER SYSTEM SET output_plugin_libraries = pgoutput, test_decoding, spock_output;" \
  -c "SELECT pg_reload_conf();"
```

<Warning>
  **The value must be unquoted here, and `pg_reload_conf()` is the whole procedure — do not restart a 1.x node to apply it.** `ALTER SYSTEM` adds the quoting for you; quoting it yourself stores a single bogus plugin literally named `"pgoutput, test_decoding, spock_output"` and the error persists (confirm with `SHOW output_plugin_libraries;` — the output must contain no quotation marks). And the pre-2.0.0 startup routine drops every replication slot that is momentarily inactive, on every peer it can reach, so restarting even one 1.x node breaks the mesh in a way that does not heal on its own. Version 2.0.0 fixes that.
</Warning>

Subscriptions move to `replicating` shortly after the reload. Check on every node:

```sql theme={null}
SELECT subscription_name, status FROM spock.sub_show_status();
```

### Upgrading from 2.x

Version 3.0.0 replaces the pgcat pooler with PgBouncer and makes the HAProxy failover tier permanent. **The client endpoint changes** from `RELEASE_NAME-pgcat` to `RELEASE_NAME-pgbouncer`; the port stays `5432`. The pgEdge nodes, their volumes and their data are kept.

<Steps>
  <Step title="Edit your values">
    Rename `pgcat:` to `pgbouncer:`, and delete `image` and `routing` from that block. Delete `proxy.enabled`. The chart refuses to render while a `pgcat` key, a pgcat image, `routing` or `proxy.enabled: false` remains, so an upgrade with stale values stops before anything changes:

    ```text theme={null}
    pgedge 3.0.0: the pgcat pooler was REPLACED by PgBouncer and the `pgcat` values key was removed.
    ```
  </Step>

  <Step title="Optionally shorten the client gap">
    Point applications at `RELEASE_NAME-pgedge-proxy.GVC_NAME.cpln.local:5432` first. That is HAProxy, present on any 2.2.0 install with the default `proxy.enabled: true`. It is unpooled, so keep the connection count well below what each node accepts.
  </Step>

  <Step title="Upgrade the release">
    ```bash theme={null}
    cpln helm upgrade RELEASE_NAME oci://ghcr.io/controlplane-com/templates/pgedge \
      --version 3.0.0 --gvc GVC_NAME -f values.yaml
    ```

    Every pgEdge node restarts once, one at a time within each location — with a single node per location, that is a write interruption of about a minute. The `RELEASE_NAME-pgcat` workload is replaced by `RELEASE_NAME-pgbouncer`, and a release that ran with `proxy.enabled: false` gains the HAProxy tier.
  </Step>

  <Step title="Point applications at the new endpoint">
    `RELEASE_NAME-pgbouncer.GVC_NAME.cpln.local:5432`.
  </Step>

  <Step title="Take a backup">
    Backups change format in 3.0.0 (one database, `pg_dump` custom format, `.dump`), and the restore job reads only the new format. Run a backup right after upgrading — see [Backing Up](#backing-up).
  </Step>
</Steps>

`pgcat.routing: single-writer` has no equivalent: every location now writes to its own node, which was already the default from 2.1.0. A release still on 2.0.x routed every write to the first location's first node, so if your application depends on a single write target, plan for multi-master writes before upgrading.

### Scaling and Availability

**What each failure needs.** Your application never has to pick another node for anything inside a location: the template handles that, and the application needs only the ordinary retry described below. The one thing the template cannot handle for you is the loss of a whole location, because the pooled endpoint always reaches the PgBouncer in the client's own location and PgBouncer has no per-location address.

| Failure | Handled by | What it needs |
| - | - | - |
| A pgEdge node | The template (the [HAProxy failover tier](#haproxy-failover-tier) moves to another local node, or to a remote location if every local node is down) | 3 nodes per location in production. New connections fail for about 3 seconds while HAProxy switches |
| A PgBouncer or HAProxy replica | The template | 2 replicas of each per location (the default) |
| An upgrade or restart of the nodes | The template (nodes restart one at a time within a location; HAProxy carries traffic) | At least 2 nodes per location. New connections fail for a few seconds when a location's `replica-0` restarts; with 1 node per location, writes stop for about a minute |
| A whole location | The template for the database — the other locations keep serving; **your application** for its own clients | At least 2 locations, and one of: **your application running in at least 2 of them** behind a public endpoint that spans them — a workload's [canonical endpoint](/reference/workload/general#canonical-endpoint-global) already sends each request to the nearest healthy location; or, for an application in the location whose database tier is down, **connecting directly to a node in another location** at `replica-N.RELEASE_NAME-pgedge.LOCATION_NAME.GVC_NAME.cpln.local:5432` |

A direct connection to a remote node is unpooled and counts against that node's 300 connections, of which the poolers can use up to `defaultPoolSize × maxReplicas × number of locations`. Keep the application's own pool small, and switch back to the pooled endpoint once its location recovers.

**Retry logic your application needs.** Every row above can still fail a connection or a statement for a moment:

* **Retry failed connections** with a short backoff (about 0.5–1 seconds) for 10–15 seconds. New connections fail for a few seconds while HAProxy moves off a node, and a PgBouncer replica that dies drops the connections going through it; new connections go to the other replica straight away.
* **Retry a failed statement**, and make retried writes idempotent: a statement running on a process that fails returns an error, and may already have committed.

Spock replicates asynchronously, so writes a lost location had not yet replicated reach the others when it returns.

**Nodes per location.** Use at least 3 nodes per location in production so the failover tier has a local node to fail over to; a single node per location fails over to a remote location instead. All nodes in all locations are writable at all times.

**Failback.** When a failed node recovers, connections PgBouncer already holds keep using the failover target until they are recycled, so traffic returns to the local node within about 5 minutes rather than instantly.

**Upgrades and restarts.** A change that restarts the node workload restarts the nodes one at a time within each location, while locations restart in parallel. With at least 2 nodes per location the failover tier carries traffic throughout; with a single node per location every location loses its node together, so plan a write interruption of about a minute. Data is preserved on each node's volume, and the mesh reconciles itself on boot.

<Warning>
  **Self-repair restores replication, not history.** When a broken subscription is rebuilt, replication resumes from the current position — rows written on a node while its subscription was down are **not** backfilled, and a node that took writes while cut off needs those rows reconciled by hand.
</Warning>

**Growing or shrinking the cluster.** Changing `locations` after data exists is not a verified path. The template creates subscriptions without initial synchronization, so a node added to a cluster that already holds data starts without its tables and rows and receives only changes made after it joins. Plan the final topology before loading data, or rebuild through a new release and a [restore](#restoring-a-backup). If you remove a location from the GVC, remove it from `locations` in the same change; otherwise every node logs a `WARNING` about the missing location on each restart.

**Seeing which node serves a location.** HAProxy's stats page shows which node is active, which are backups, and which are `UP` or `DOWN`:

```bash theme={null}
cpln port-forward RELEASE_NAME-pgedge-proxy 8405:8405 --gvc GVC_NAME
```

Then open `http://localhost:8405/stats` in a browser.

**Storage.** Each node's volume starts at `volumeset.capacity` GiB. Enable `volumeset.autoscaling` to grow volumes automatically up to `maxCapacity`; the volume set is never shrunk by the template.

### Troubleshooting

<AccordionGroup>
  <Accordion title="The deployment never starts and cpln logs returns nothing">
    **Symptom:** no replica ever becomes ready, and `cpln logs` for the node, PgBouncer or HAProxy workload returns zero lines.

    **Cause:** the credentials secret named by `postgres.credentialsSecretName` does not exist, so the container is never started and has nothing to log.

    **Fix:** read the message, which names the missing secret:

    ```bash theme={null}
    cpln workload get-deployments RELEASE_NAME-pgedge --gvc GVC_NAME -o yaml
    ```

    Create the secret as described in [Prerequisites](#prerequisites). The deployment recovers on its own within several minutes, or run `cpln workload force-redeployment RELEASE_NAME-pgedge --gvc GVC_NAME`.
  </Accordion>

  <Accordion title="Nodes restart with FATAL locations declared in values are not in GVC">
    **Symptom:** the install succeeds, but the node containers exit and restart, and the node logs show:

    ```text theme={null}
    [pgedge] FATAL: locations declared in values are not in GVC 'GVC_NAME': aws-eu-central-1
    [pgedge] GVC 'GVC_NAME' has: aws-us-west-2 aws-us-east-2
    ```

    **Cause:** `locations` names a location the GVC does not have. The platform does not validate this at install, so the nodes check it themselves at startup and refuse to initialize a cluster whose declared members can never exist.

    **Fix:** either add the location to the GVC or remove it from `locations`, then upgrade the release. Read the lines with a server-side filter:

    ```bash theme={null}
    cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-pgedge"} |= "[pgedge]"' --limit 50 --since 10m
    ```

    A node that has **already** initialized logs `[pgedge] WARNING: locations declared in values are not in GVC` instead and keeps serving, so this check never takes down a running cluster. A line reading `[pgedge] WARNING: could not read the location list of GVC` means the check could not run at all and was skipped — verify the GVC's locations yourself with `cpln gvc get GVC_NAME -o yaml`.
  </Accordion>

  <Accordion title="helm upgrade fails with the pgcat pooler was REPLACED by PgBouncer">
    **Symptom:** an upgrade to 3.0.0 stops at render with `pgedge 3.0.0: the pgcat pooler was REPLACED by PgBouncer`, `proxy.enabled was REMOVED`, `pgbouncer.routing does not exist` or `pgbouncer.image is a pgcat image`.

    **Cause:** your values file still carries 2.x keys. Nothing was changed — the chart refuses before any resource is touched.

    **Fix:** edit the values as described in [Upgrading from 2.x](#upgrading-from-2-x) and run the upgrade again.
  </Accordion>

  <Accordion title="Connections fail with unsupported startup parameter in options">
    **Symptom:** a client that sets `options` (for example `PGOPTIONS='-c some_setting=value'`) is refused at connect with `unsupported startup parameter in options`.

    **Cause:** PgBouncer passes only a fixed list of settings through `options`; see [PgBouncer Connection Pooler](#pgbouncer-connection-pooler).

    **Fix:** remove the setting from `options` and set it with `ALTER ROLE … SET` on every node, or with `SET` at the start of the session in `session` mode.
  </Accordion>

  <Accordion title="CREATE TABLE fails with cannot be added to replication set default">
    **Symptom:**

    ```text theme={null}
    ERROR:  table events cannot be added to replication set default
    DETAIL:  table does not have PRIMARY KEY and given replication set is configured to replicate UPDATEs and/or DELETEs
    ```

    **Cause:** the template adds every new table to the `default` replication set, which replicates `UPDATE` and `DELETE`; Spock cannot do that for a table without a primary key.

    **Fix:** give the table a `PRIMARY KEY` — a `uuid` column with `DEFAULT gen_random_uuid()` is the recommended shape (see [Primary Keys](#primary-keys)).
  </Accordion>

  <Accordion title="A table exists on one node only or rows written on one node never appear on the others">
    **Symptom:** a table you created is missing on other nodes, or the other nodes' subscriptions stop at `down` after you created it.

    **Cause:** DDL does not replicate. A plain `CREATE TABLE` applied to one node only, or `spock.replicate_ddl()` was run without the follow-up `spock.repset_add_table()` on the other nodes.

    **Fix:** follow [Schema Changes and DDL](#schema-changes-and-ddl) — create the table on every node directly, or broadcast it and then add it to the replication set on every other node. Confirm with `SELECT * FROM spock.tables WHERE relname = 'TABLE_NAME';` on each node.
  </Accordion>

  <Accordion title="spock.sub_show_status() shows subscriptions down">
    **Symptom:** `SELECT subscription_name, status FROM spock.sub_show_status();` returns `down` for one or more rows, and writes made on one node do not appear on another.

    **Causes and fixes, in order of likelihood:**

    * **A node that is still booting or restarting** — the mesh reconciles itself on boot. Give it a few minutes and re-check; rows written while a subscription was down are not backfilled (see [Scaling and Availability](#scaling-and-availability)).
    * **A table created on one node only** — see the entry above.
    * **A 1.x data directory** — the server log (not `cpln logs`) shows `library "spock_output" may not be used as an output plugin`. Apply the fix under [Switching Replication On for an Existing 1.x Cluster](#switching-replication-on-for-an-existing-1-x-cluster).
  </Accordion>

  <Accordion title="A restore fails with already has objects or no backups of release">
    **Symptom:** `cpln workload cron get` shows the restore run `failed`, and the backup job's log reads one of:

    ```text theme={null}
    [pgedge-backup] FATAL: node replica-0.RELEASE_NAME-pgedge.LOCATION_NAME.GVC_NAME.cpln.local already has 10 object(s) (tables, views, sequences, functions, types or schemas) in 'mydb'. Restore only into an empty database: install a new release, or drop them on EVERY node first.
    [pgedge-backup] FATAL: no backups of release 'RELEASE_NAME' (pgedge-RELEASE_NAME-*.dump) under aws://BUCKET/PREFIX/ -- set RESTORE_FILE to restore another release's backup
    ```

    **Cause:** the restore runs only into a database that is empty on every node, and without `RESTORE_FILE` it picks only this release's own backups. It refused before changing anything.

    **Fix:** drop your objects on every node as described in [Restoring a Backup](#restoring-a-backup), or set `RESTORE_FILE` to the backup you want — for another release's backup, the path inside the bucket.
  </Accordion>

  <Accordion title="PgBouncer exits with database name pgbouncer is reserved">
    **Symptom:** the `RELEASE_NAME-pgbouncer` replicas restart, and their log shows `[pgbouncer] FATAL: database name 'pgbouncer' is reserved for PgBouncer's admin console; use another name in your credentials secret`.

    **Cause:** the `database` entry of your credentials secret is `pgbouncer`.

    **Fix:** use a different database name. The nodes create the database on first boot, so on an install that has already initialized, reinstall under a new release with the corrected secret.
  </Accordion>
</AccordionGroup>

## Important Notes

* **Your application must retry failed connections and statements** — without it, every upgrade and replica restart shows up as errors in your app. Retry with a short backoff for 10–15 seconds and make retried writes idempotent; see [Scaling and Availability](#scaling-and-availability).
* **3.0.0 changes the client endpoint** to `RELEASE_NAME-pgbouncer`, and backups change format — take a backup right after upgrading. See [Upgrading from 2.x](#upgrading-from-2-x).
* **Never run `helm upgrade` on a 1.x release against any later version** — it deletes the GVC the 1.x chart created and everything in it. Migrate to a new release instead: [Upgrading from 1.x](#upgrading-from-1-x).
* **The GVC must already contain every location you list**, and may contain more. A missing location is not caught at install — the nodes exit with `[pgedge] FATAL: locations declared in values are not in GVC ...`. Locations you do not list start nothing.
* **DDL does not replicate** — create every table on every node (or broadcast it and add it to the replication set on the others), and give every table a `PRIMARY KEY` or `CREATE TABLE` fails outright.
* **Use at least 3 nodes per location in production**, and at least 2 for upgrades without a write interruption.
* **Surviving a location loss needs your application in at least 2 locations, or a direct connection to a remote node** — clients reach only their own location's PgBouncer. See [Scaling and Availability](#scaling-and-availability).
* **With `internal_access.type: workload-list`, list only your clients** — the release's own workloads are added for you.
* **Fix the topology before loading data** — nodes added to a cluster that already holds data do not receive existing rows.
* **Rotating the database password is not a supported path in this version.** The password is written into each node's Spock node and subscription definitions when they are created, so changing the secret alone would break replication; plan a new release and a restore if you must rotate.
* **Release names must be unique per org** — secrets are org-wide, so two releases with the same name collide even in different GVCs.
* **Conflict resolution is last-update-wins** by commit timestamp. Route writes for a given entity to one node in your application if you need stronger guarantees.

## External References

<CardGroup cols={2}>
  <Card title="pgEdge Documentation" icon="book" href="https://docs.pgedge.com/">
    Official pgEdge Distributed PostgreSQL documentation
  </Card>

  <Card title="Spock Documentation" icon="book" href="https://docs.pgedge.com/spock-v5/">
    Spock multi-master replication extension, including the SQL function reference
  </Card>

  <Card title="PgBouncer Configuration" icon="book" href="https://www.pgbouncer.org/config.html">
    Pool modes, connection limits and startup parameters
  </Card>

  <Card title="PgBouncer Usage and Admin Console" icon="terminal" href="https://www.pgbouncer.org/usage.html">
    Admin console commands such as SHOW POOLS and RECONNECT
  </Card>

  <Card title="pgEdge Template" icon="github" href="https://github.com/controlplane-com/templates/tree/main/pgedge">
    View the source files, default values, and chart definition
  </Card>
</CardGroup>