CockroachDB

​

Overview

​

What Gets Created

​

Prerequisites

cpln gvc get GVC_NAME -o yaml

[cockroach] FATAL: locations declared in values are not in GVC 'GVC_NAME': aws-us-west-2
[cockroach] GVC 'GVC_NAME' has: aws-us-east-1 aws-us-east-2

cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-cockroach"} |= "[cockroach]"' --limit 50 --since 10m

​

Installation

cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/cockroach \
  --version 2.0.0 --gvc GVC_NAME

​

Configuration

​

Locations

# ─── Locations ────────────────────────────────────────────────────────────────
# This chart deploys into the GVC you install into — it does NOT create one.
# Every location listed here MUST already exist in that GVC, or the CockroachDB
# nodes refuse to initialise with a named error (see Prerequisites in the README).
# Extra locations in the GVC are fine: nothing CockroachDB-related runs in them.
#
# Surviving the loss of a whole location needs at least THREE locations. One
# location with three replicas survives a node loss, not a region loss.
locations:
  - name: aws-us-east-1
    replicas: 3

locations:
  - name: aws-us-east-1
    replicas: 3
  - name: aws-us-east-2
    replicas: 3
  - name: aws-us-west-2
    replicas: 3

• replicas: 0 is refused. To stop running in a location, remove it from locations — a zero-replica location would still be added to the database as a region that no node ever joins.
• A one-node deployment is refused. Each node joins every other node, so a single node has nothing to join and cannot start. Use at least 3 replicas in total; 3 in one location is the smallest supported shape, matching CockroachDB’s default replication factor of 3.

​

Image and Multi-Zone Placement

image: cockroachdb/cockroach:v25.4.0

# Confirm every location in your GVC supports multi-zone before enabling this --
# a location that does not will wedge with no error. See the README.
multiZone: false

​

Resources

resources:
  cpu: 2
  memory: 4Gi

​

Database

database:
  name: mydb
  user: myuser

​

Storage

volumeset:
  capacity: 10 # Initial capacity in GiB (minimum is 10)
  autoscaling:
    enabled: false # Set to true to enable autoscaling
    maxCapacity: 100 # Maximum capacity in GiB when autoscaling is enabled
    minFreePercentage: 10 # Minimum free percentage to trigger scaling when autoscaling is enabled
    scalingFactor: 1.2 # Scaling factor to determine how much to scale up when autoscaling is triggered

​

Internal Access

internal_access:
  type: same-gvc # options: same-gvc, same-org, workload-list; used for CockroachDB when pgbouncer is disabled
  workloads:  # Note: can only be used if type is workload-list
    #- //gvc/GVC_NAME/workload/WORKLOAD_NAME

​

PgBouncer

pgbouncer:
  enabled: true
  image: edoburu/pgbouncer:v1.25.1-p0
  poolMode: transaction  # options: session, transaction, statement
  defaultPoolSize: 25    # number of real CockroachDB connections PgBouncer maintains per pod
  maxClientConn: 250     # maximum number of client connections PgBouncer accepts per pod
  maxDbConnections: 100  # hard cap on total CockroachDB connections regardless of how many PgBouncer pods are running
  minReplicas: 2         # per location
  maxReplicas: 4         # per location
  serverCheckDelay: 30      # seconds between idle server connection health checks (default 30)
  serverConnectTimeout: 2   # seconds before giving up on a new server connection (default 15)
  serverLoginRetry: 0       # seconds before retrying a failed server login; 0 = no caching of failures (default 15)
  clientLoginTimeout: 10    # seconds before rejecting a client waiting for login (default 60)
  queryWaitTimeout: 10      # seconds before rejecting a logged-in client waiting for a server connection (default 120)
  internal_access:
    type: same-gvc # options: same-gvc, same-org, workload-list; controls who can connect to PgBouncer
    workloads:  # Note: can only be used if type is workload-list
      #- //gvc/GVC_NAME/workload/WORKLOAD_NAME
  resources:
    maxCpu: 200m
    minCpu: 100m
    maxMemory: 1Gi
    minMemory: 128Mi

• maxDbConnections is a hard cap on real CockroachDB connections shared across all PgBouncer replicas — set it to what your cluster can safely accept regardless of how many replicas are running.
• minReplicas / maxReplicas apply per location; PgBouncer autoscales on request rate between them.
• resources names the limit maxCpu / maxMemory beside the reservation minCpu / minMemory. The pre-2.0.0 names cpu and memory are refused at render with a message naming the replacement.
• The remaining timeouts are PgBouncer’s own settings, in seconds — see the PgBouncer configuration reference.

​

Backup

backup:
  enabled: false
  image: ghcr.io/controlplane-com/backup-images/cockroach-backup:1.1
  schedule: "0 2 * * *"
  activeDeadlineSeconds: 14400   # hard kill after 4 hours if backup hangs
  location: aws-us-east-1        # MUST be one of `locations` above; put it near your bucket
  resources:
    cpu: 500m
    memory: 512Mi
  provider: aws  # options: aws, gcp
  aws:
    bucket: my-backup-bucket
    region: us-east-1
    cloudAccountName: my-backup-cloudaccount
    policyName: my-backup-policy
    prefix: cockroach/backups
  gcp:
    bucket: my-backup-bucket
    cloudAccountName: my-backup-cloudaccount
    prefix: cockroach/backups

​

Connecting

postgresql://myuser@RELEASE_NAME-cockroach-pgbouncer.GVC_NAME.cpln.local:5432/mydb?sslmode=disable

cpln workload exec RELEASE_NAME-cockroach --gvc GVC_NAME \
  --replica RELEASE_NAME-cockroach-0 --container cockroach \
  -- cockroach node status --insecure --host=localhost:26257

cpln port-forward RELEASE_NAME-cockroach 8080:8080 --gvc GVC_NAME

​

Operations

​

Backing Up

BACKUP INTO 's3://BUCKET_NAME/PREFIX?AUTH=implicit&AWS_REGION=BUCKET_REGION' AS OF SYSTEM TIME '-10s';

• It is a full-cluster backup, not a backup of mydb alone: every database, every table, users and cluster settings land in one backup collection at BUCKET_NAME/PREFIX/. Each run adds a new full backup to that collection; nothing deletes older ones, so manage retention with a bucket lifecycle rule.
• The CockroachDB nodes upload the data themselves, authenticating with the workload identity’s cloud account binding (AUTH=implicit). The cron job only triggers the statement, which is why the cloud permissions are on RELEASE_NAME-cockroach-identity and the restore statements later on this page run from a node.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:ListBucket",
                "s3:GetObjectVersion",
                "s3:DeleteObjectVersion",
                "s3:GetBucketLocation",
                "s3:AbortMultipartUpload",
                "s3:ListBucketMultipartUploads",
                "s3:ListMultipartUploadParts"
            ],
            "Resource": [
                "arn:aws:s3:::YOUR_BUCKET_NAME",
                "arn:aws:s3:::YOUR_BUCKET_NAME/*"
            ]
        }
    ]
}

cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-cockroach-backup"}' --limit 50 --since 24h

cpln workload exec RELEASE_NAME-cockroach --gvc GVC_NAME \
  --replica RELEASE_NAME-cockroach-0 --container cockroach \
  -- cockroach sql --insecure --host=localhost:26257 \
  --execute="SHOW BACKUPS IN 's3://BUCKET_NAME/PREFIX?AUTH=implicit&AWS_REGION=BUCKET_REGION';"

​

Restoring a Backup

cpln workload exec RELEASE_NAME-cockroach --gvc GVC_NAME \
  --replica RELEASE_NAME-cockroach-0 --container cockroach \
  -- cockroach sql --insecure --host=localhost:26257 \
  --execute="RESTORE DATABASE mydb FROM LATEST IN 's3://BUCKET_NAME/PREFIX?AUTH=implicit&AWS_REGION=BUCKET_REGION' WITH new_db_name = 'mydb_restored';"

cpln workload exec RELEASE_NAME-cockroach --gvc GVC_NAME \
  --replica RELEASE_NAME-cockroach-0 --container cockroach \
  -- cockroach sql --insecure --host=localhost:26257 \
  --execute="DROP DATABASE mydb CASCADE; DROP USER myuser;"

cpln workload exec RELEASE_NAME-cockroach --gvc GVC_NAME \
  --replica RELEASE_NAME-cockroach-0 --container cockroach \
  -- cockroach sql --insecure --host=localhost:26257 \
  --execute="RESTORE FROM LATEST IN 's3://BUCKET_NAME/PREFIX?AUTH=implicit&AWS_REGION=BUCKET_REGION';"

​

Upgrading from 1.x

Error: execution error at (cockroach/templates/identity.yaml:2:4): cockroach 2.0.0: the `gvc` values
key was REMOVED. This chart no longer creates a GVC -- it deploys into the GVC you install into, and
`gvc.locations` moved to the top-level `locations`. DO NOT `helm upgrade` a 1.x release onto 2.0.0 ...

cpln helm uninstall RELEASE_NAME --gvc GVC_NAME

​

Scaling and Availability

• Node loss. With three nodes in a location (the default), CockroachDB’s replication factor of 3 keeps every range available through the loss of one node. Scale a location by raising locations[].replicas and upgrading; new nodes join the cluster on their own.
• Location loss needs three or more locations. With three or more entries in locations, the first deploy registers every location as a CockroachDB region and sets the survival goal of database.name to SURVIVE REGION FAILURE, so the cluster tolerates the loss of an entire location. With one or two locations that step is skipped — no CockroachDB setting can make a one- or two-location cluster survive a region loss. Verify what your cluster actually has:
  cpln workload exec RELEASE_NAME-cockroach --gvc GVC_NAME \
    --replica RELEASE_NAME-cockroach-0 --container cockroach \
    -- cockroach sql --insecure --host=localhost:26257 \
    --execute="SHOW SURVIVAL GOAL FROM DATABASE mydb; SHOW REGIONS FROM CLUSTER;"
  
  If the region setup did not complete, the start log of the first node says so explicitly (ERROR: Multi-region configuration is INCOMPLETE) and prints SHOW REGIONS FROM CLUSTER; see Troubleshooting.
• Rolling restarts during an outage are not covered. A cluster that is already missing a location is at its fault-tolerance limit; rolling out or restarting nodes in the remaining locations during that outage causes a brief period of unavailability for ranges on those nodes. Wait for the lost location to return before upgrading.
• Your application must retry. PgBouncer routes around failed nodes, but transient errors still reach the client during failover events — a location outage or a rolling restart — while PgBouncer cycles through backends and leader elections complete. Without retries those errors propagate to your users.
• Removing a location from locations does not drain its nodes. They refuse to start rather than rejoining from a region the database was not told about. Decommission the nodes in that location first if they hold data.
• PgBouncer scales on its own between pgbouncer.minReplicas and pgbouncer.maxReplicas in each location; raise maxReplicas when the pooler, not the database, becomes the bottleneck.
• Data survives restarts and upgrades, not an uninstall: cpln helm uninstall removes the volume set with the release, taking a final snapshot retained for 7 days. Take a backup first if the data matters.

​

Troubleshooting

[cockroach] FATAL: locations declared in values are not in GVC 'GVC_NAME': aws-us-west-2
[cockroach] GVC 'GVC_NAME' has: aws-us-east-1 aws-us-east-2
[cockroach] Refusing to initialise a cluster whose declared regions can never exist.

​

Important Notes

• Every location in locations must already exist in the GVC. A missing one is accepted at install and refused by the nodes at boot; a GVC location you did not list runs nothing.
• Never helm upgrade a 1.x release onto 2.0.0. It deletes the GVC the 1.x release created and everything in it. Install a new release and move the data — see Upgrading from 1.x.
• Surviving a region loss needs at least three locations. The default single location survives a node failure, not a location failure.
• The cluster runs in insecure mode. There are no SQL credentials; internal_access, pgbouncer.internal_access and the GVC boundary are the only access control.
• Confirm multi-zone support in every GVC location before setting multiZone: true. An unsupported location wedges the deployment with no log output.
• Use at least 3 replicas in total. A one-node deployment is refused at render; replicas: 0 on a location is refused too — remove the location instead.
• Removing a location does not drain its nodes. Decommission them first if they hold data.
• Backups are full-cluster backups. Restoring into a live cluster must use RESTORE DATABASE mydb ... WITH new_db_name, never a bare RESTORE FROM LATEST IN — see Restoring a Backup.
• backup.location must be one of locations. The cron runs only there; anywhere else it is suspended.
• Your application must retry transient errors. Failover events surface them through PgBouncer.
• Access-knob changes take up to a few minutes to apply. Re-test before concluding a change did not work.
• cpln helm uninstall deletes the volume set and with it the data, after a final snapshot retained 7 days. Back up first.

​

External References