etcd Multi-Location

Overview

etcd is a strongly consistent key-value store used as the coordination layer for leader election, distributed locking, and shared configuration. This template deploys a single stretched etcd cluster with exactly one member per Control Plane location, sharing one Raft quorum across regions, with Raft timers tuned for cross-region round trips and auto-compaction enabled so the backend does not grow without bound. It is also the consensus store that Postgres Multi-Location bundles for Patroni, and is independently useful as a cross-region coordination store for your own services.

This template deploys into an existing GVC you already have; it does not create, provision or manage one. The GVC must have at least two locations, and every location you list in global.locations must already be one of them. A GVC location you do not list runs nothing — its deployment reads This workload location is deactivated because maxScale is set to 0. For a cluster in a single location use the etcd template instead.

What Gets Created

Resource	Name	Purpose	Optional
Workload	`RELEASE_NAME-etcd`	The etcd members: a stateful workload with exactly one replica in each location listed in `global.locations`; client API on port `2379`, peer traffic on port `2380`	—
Volume Set	`RELEASE_NAME-etcd-vs`	`/var/lib/etcd` on every member (Raft write-ahead log and snapshots) — `ext4` on general-purpose SSD, snapshots retained 7 days, final snapshot on uninstall	—
Secret	`RELEASE_NAME-etcd-startup`	The startup script that derives each member’s name, peer URL and full cluster list at boot from the location it runs in	—
Identity	`RELEASE_NAME-etcd-identity`	Bound to the workload so it can read the startup script and its own GVC	—
Policy	`RELEASE_NAME-etcd-policy`	Grants the identity `reveal` on the startup secret and nothing else	—
Policy	`RELEASE_NAME-etcd-gvc-policy`	Grants the identity `view` on exactly the one GVC the release is installed into, so each member can confirm at boot that the GVC really has every declared location	—

The template creates no credential secret: etcd here runs without TLS or authentication, and access is governed by the internal firewall (see Internal Access). Every member is named RELEASE_NAME-etcd-LOCATION, so etcdctl member list maps one-to-one onto your location list.

Prerequisites

A GVC with at least two locations, covering every entry in global.locations

This template deploys into an existing GVC and does not create one. Read the GVC’s locations and compare them with the list you intend to install with:

cpln gvc get GVC_NAME -o yaml

spec.staticPlacement.locationLinks is the list. Every name you put in global.locations must appear there — a declared location the GVC lacks makes a fresh member refuse to start (see Troubleshooting). Three locations is the recommended production shape; see Locations and Quorum.

Nothing else is needed: no secret to create, no cloud account, no bucket, no custom domain.

Installation

The default global.locations list (aws-us-east-1, aws-us-west-2) is only the smallest cluster the chart allows — set it to your GVC’s own locations in a values file, then install:

# etcd-locations.yaml — one etcd member per location; every name must already be in the GVC
global:
  locations:
    - name: LOCATION_1
    - name: LOCATION_2
    - name: LOCATION_3

cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/etcd-multi-location \
  --version 2.0.0 --gvc GVC_NAME -f etcd-locations.yaml

To install with another method, follow the instructions for your preferred tool:

UI

Browse, install, and manage templates visually

CLI

Manage templates from your terminal

Terraform

Declare templates in your Terraform configurations

Pulumi

Declare templates in your Pulumi programs

Configuration

Each subsection below shows the shipped default for one area of values.yaml. Override only what you need with a values file or --set.

Locations and Quorum

global:
  # One etcd member per location. Minimum 2; 3 survives losing one location,
  # 5 survives losing two. See the quorum table in the README — 3 is the
  # recommended production shape. This default is the smallest cluster the
  # chart allows, because no location list can be right for every GVC: set it
  # to YOUR GVC's locations before installing.
  locations:
    - name: aws-us-east-1
    - name: aws-us-west-2

global.locations — The member map: one etcd member is deployed in each listed location, and at least two locations are required. Every name must already be a location of the GVC you install into. Each location runs exactly one member; a replicas value other than 1 fails the render in a standalone install.

The list lives under global so a parent chart — Postgres Multi-Location bundles this template as a subchart — sets it once and Helm propagates it here; the two lists can then never be edited apart. etcd commits a write only when a majority of members accept it. Because there is exactly one member per location, the location is the failure domain:

Locations	Majority needed	Location losses survived	What that means
2	2	0	Losing either one stops writes. The survivor holds current data, but recovery is manual.
3	2	1	Automatic failover. The recommended shape.
4	3	1	No better than 3, and costs more.
5	3	2	Survives losing two locations.

With N locations you survive floor((N-1)/2) losses, so an even count never buys anything over the odd count below it. Two locations is permitted as a deliberate warm-standby topology, but it survives nothing automatically.

Changing global.locations on a running release reprovisions the cluster: every member restarts with a new cluster list. This is not etcd’s graceful member add / member remove path, so plan it as a maintenance window — and change the GVC’s locations and this list together.

Image and Resources

image: controlplanecorporation/etcd:0.1

resources:
  cpu: 500m
  memory: 512Mi

image — The etcd container image. The shipped tag carries etcd 3.6.5 together with etcdctl and etcdutl.
resources.cpu / resources.memory — CPU and memory limit for each member. See the etcd hardware recommendations for sizing guidance.

Raft Tuning

# Raise both if your locations are more than ~250 ms apart.
tuning:
  heartbeatIntervalMs: 250 # ~0.5-1.5x the worst round trip between locations
  electionTimeoutMs: 5000 # must be >= 10x heartbeatIntervalMs; maximum 50000

tuning.heartbeatIntervalMs — Leader heartbeat interval, roughly 0.5–1.5× the worst round trip between your locations.
tuning.electionTimeoutMs — How long a follower waits without a heartbeat before campaigning. Must be at least 10× the heartbeat interval and at most 50000; both bounds are enforced at render time.

The defaults suit locations within a few hundred milliseconds of each other, such as a US ↔ Europe spread. Locations further apart (US ↔ Asia-Pacific) need both values raised in proportion; see Tuning for Latency.

Compaction and Backend Growth

etcd never discards a superseded revision on its own. Every write creates a new revision and the old one is kept until a compaction removes it, so the backend grows with time alone whenever a client writes on a timer — Patroni, for example, renews its leader lease every few seconds, producing new revisions whether or not any application data changes. Left uncompacted, an otherwise idle cluster reaches etcd’s default 2 GiB backend quota in a matter of months. When the quota is reached, etcd raises a cluster-wide NOSPACE alarm and every member goes read-only until an operator intervenes. Auto-compaction is what prevents that, and this template configures it by default:

tuning:
  # etcd keeps every superseded revision until told otherwise, so compaction is
  # required rather than optional and cannot be switched off here.
  autoCompactionMode: periodic # periodic (retention is a duration) or revision (a revision count)
  autoCompactionRetention: 1h # periodic needs an explicit unit (1h, 30m, 24h); revision takes a count
  quotaBackendBytes: 0 # backend size limit in bytes; 0 = etcd's own default of 2 GiB

tuning.autoCompactionMode — How autoCompactionRetention is read. periodic treats it as a time window; revision treats it as a number of revisions to keep. There is no “off” — an unrecognized mode is rejected at render time.
tuning.autoCompactionRetention — How much history is kept before old revisions are discarded. The default 1h keeps the backend flat. Raising it to 24h or more buys a longer history window at the cost of a larger backend.
tuning.quotaBackendBytes — Backend size ceiling in bytes. 0 (the default) means the flag is not passed at all and etcd applies its own 2 GiB limit. Leave it at 0 unless the keyspace genuinely outgrows that; 8 GiB (8589934592) is etcd’s own suggested maximum, above which it warns at startup.

In periodic mode a retention value without a unit means hours. etcd reads a bare 30 as thirty hours, not thirty minutes. This template rejects an unsuffixed value at render time rather than letting it become a silently wrong window — write 30m, 1h or 24h.

quotaBackendBytes takes a plain byte count, not a size suffix. etcd itself refuses --quota-backend-bytes 2Gi at boot and the cluster would crash-loop, so a 2Gi-style value is rejected at render time instead. Write 2147483648.

Compaction is deliberately not switchable off: a retention of 0, an unrecognized mode and a negative quota (which etcd reads as “no quota at all”) are each rejected at render time, because each one produces a chart that installs cleanly and fails weeks later.

Compaction frees pages for reuse inside the backend file; it does not shrink the file. The reported DB SIZE therefore plateaus rather than dropping, while DB SIZE IN USE falls back to the size of the live keyspace. That is expected and is sufficient to stay under the quota, because reclaimed pages are reused for new writes. Only etcdctl defrag returns space to the filesystem, and this template does not automate it — defragmentation blocks the member it runs on.

etcdctl endpoint status reports QUOTA | 0 B when quotaBackendBytes is 0, because that column reflects the flag rather than the effective limit. The effective quota in that case is still etcd’s 2 GiB default, which etcd logs at startup as enabled backend quota with default value.

Storage

volumeset:
  capacity: 10 # initial capacity in GiB (minimum is 10)

volumeset.capacity — Size in GiB of the persistent volume each member gets for /var/lib/etcd (minimum 10). The volume set keeps snapshots for 7 days and takes a final snapshot when the release is uninstalled.

Internal Access

internalAccess:
  type: same-gvc # options: same-gvc, same-org, workload-list
  workloads: [] # only used when type is workload-list; the etcd workload is added automatically
  #- //gvc/GVC_NAME/workload/WORKLOAD_NAME

internalAccess.type — Controls which workloads can reach the client API on port 2379. Traffic between locations inside one GVC is same-GVC traffic, so the default covers a stretched cluster with no extra rule.

Type	Description
`same-gvc`	Allow access from all workloads in the same GVC (recommended)
`same-org`	Allow access from all workloads in the same organization
`workload-list`	Allow access only from the specific workloads listed in `internalAccess.workloads`

internalAccess.workloads — The workload links allowed in when type is workload-list. List only your clients. The members of this workload must also be able to reach each other, so the template appends its own workload link to the list for you; a client that is not listed is still refused, and the cluster stays whole.

etcd runs here without TLS or authentication. There is deliberately no public access, and anything the internal firewall lets in has full read and write access to the whole keyspace. If the GVC holds workloads that should not have that, use workload-list and name the clients explicitly. A firewall change can take a few minutes to take effect; re-test rather than trusting the first response.

Disaster Recovery

recovery:
  # EMERGENCY ONLY. Set to the surviving location's name to restart that member
  # as a new single-member cluster after permanently losing quorum. Read the
  # "Recovering from a lost location" section of the README first.
  forceNewClusterInLocation: ""

recovery.forceNewClusterInLocation — Leave empty in normal operation. Set it to the name of the one surviving location only after quorum is permanently lost, following Recovering From a Lost Location, and clear it again afterwards. The value must be one of the names in global.locations; anything else is rejected at render time.

Connecting

The cluster has no public endpoint: external inbound traffic is closed and no load balancer is created. Clients connect from inside the GVC, or from your machine through a port-forward.

What	Address	Notes
Client API from the same GVC	`http://RELEASE_NAME-etcd.GVC_NAME.cpln.local:2379`	Resolves to a member; the one address to give to clients
A specific location’s member	`http://replica-0.RELEASE_NAME-etcd.LOCATION.GVC_NAME.cpln.local:2379`	`LOCATION` is one of the names in `global.locations`; give a client one of these per location so it can fail over
Peer traffic	port `2380`	Used by the members themselves; clients do not need it
Credentials	none	No TLS and no authentication — access is decided by Internal Access

Always use the fully qualified hostname; the bare workload name is not reliably resolvable. Verify the cluster from inside one of its members. etcdctl is in the image and talks to the local member by default:

cpln workload exec RELEASE_NAME-etcd --gvc GVC_NAME --container etcd -- etcdctl member list -w table
cpln workload exec RELEASE_NAME-etcd --gvc GVC_NAME --container etcd -- etcdctl endpoint status --cluster -w table

member list should show one member per configured location, each named RELEASE_NAME-etcd-LOCATION, and endpoint status --cluster should show every member on the same RAFT TERM with exactly one IS LEADER: true. Then write and read a key:

cpln workload exec RELEASE_NAME-etcd --gvc GVC_NAME --container etcd -- etcdctl put /demo/key hello
cpln workload exec RELEASE_NAME-etcd --gvc GVC_NAME --container etcd -- etcdctl --command-timeout=5s get /demo/key

From another workload in the GVC, point etcdctl (or your client library) at the service address:

etcdctl --endpoints=http://RELEASE_NAME-etcd.GVC_NAME.cpln.local:2379 put hello world
etcdctl --endpoints=http://RELEASE_NAME-etcd.GVC_NAME.cpln.local:2379 get hello

To reach the cluster from your own machine, forward the client port and use a local etcdctl against http://127.0.0.1:2379:

cpln port-forward RELEASE_NAME-etcd 2379:2379 --gvc GVC_NAME

Give long-lived clients a short command timeout (etcdctl --command-timeout=5s, or the equivalent in your client library). When fewer than a majority of members are reachable, etcd makes writes wait for quorum rather than failing immediately, so a client without a timeout appears to hang.

Operations

Backing Up

Two independent layers protect the data:

Volume set snapshots. The volume set RELEASE_NAME-etcd-vs keeps snapshots for 7 days and takes a final snapshot when the release is uninstalled; this is configured by the template and needs no action. To take one on demand in a given location:
cpln volumeset snapshot create RELEASE_NAME-etcd-vs --gvc GVC_NAME --snapshot-name NAME --location LOCATION
An etcd snapshot file. etcdctl snapshot save writes a consistent, hash-protected copy of the whole keyspace from one member — any member will do, since every member holds the full keyspace. The image ships etcdctl and base64, so the file can be pulled out through cpln workload exec. Pass it through base64 on both sides — the exec stream is text-only and corrupts raw binary:
cpln workload exec RELEASE_NAME-etcd --gvc GVC_NAME --container etcd --quiet -- \ sh -c 'etcdctl snapshot save /tmp/snapshot.db >/dev/null 2>&1 && base64 /tmp/snapshot.db' \ | base64 -d > etcd-snapshot.db
Then confirm the file is intact on your machine with etcdutl, which is part of every etcd release:
etcdutl snapshot status etcd-snapshot.db -w table

The snapshot command above was checked against the shipped image’s contents and the upstream etcdctl documentation, but has not been exercised against a live install of this template. Run it once on a test release and confirm etcdutl snapshot status reports the hash before you rely on it.

Restoring a Backup

Restoring is not a procedure this template performs or has verified. The upstream path — etcdutl snapshot restore — rebuilds a data directory for every member with a new cluster ID and requires each member to be stopped and started from the rebuilt directory. This template manages the data directory through its startup script and gives you no supported way to stop a member and replace /var/lib/etcd on its volume. Restoring a volume set snapshot in place is likewise unverified for this template. If you must recover a keyspace, follow the etcd disaster recovery guide against a new release and rehearse it on a throwaway release first. For the loss of a minority or a majority of locations with the surviving members intact, see Recovering From a Lost Location instead — that path needs no snapshot. Where etcd came bundled with Postgres Multi-Location as its Patroni consensus store, consult that page before attempting any etcd-level recovery.

Migrating From Version 1

Versions before 2.0.0 created their own GVC; 2.0.0 deploys into one you already have.

Do not cpln helm upgrade a 1.x release onto 2.0.0. Once the chart stops declaring a GVC, Helm removes the one the old release created — and deleting a GVC deletes every workload, volume set and identity inside it. The chart refuses to render when your values still carry the 1.x global.gvc key, so a values-carrying upgrade fails safely before any resource is touched; that guard cannot fire on an upgrade run with no values at all. The procedure below is the safety; the render guard is only a backstop.

Install 2.0.0 as a new release against an existing GVC, move the keyspace across, then remove the old release.

Choose the target GVC

Pick a GVC whose locations are exactly the ones you want members in, and read them back with cpln gvc get GVC_NAME -o yaml — spec.staticPlacement.locationLinks is the list. Do not point the new release at the GVC the old release created: that GVC is still owned by the old Helm release, and uninstalling it later would take the new workloads with it.

Export the old cluster's keyspace

cpln workload exec OLD_RELEASE_NAME-etcd --gvc OLD_GVC_NAME --container etcd --quiet \
  -- etcdctl get "" --prefix -w json > keyspace.json

Alternatively take an etcdctl snapshot save file as shown in Backing Up, if you prefer a hash-protected copy to keep.

Install 2.0.0 as a new release

Replace global.gvc.name and global.gvc.locations in your old values with a single top-level global.locations list naming the chosen GVC’s locations, then install into that GVC as shown in Installation.

Confirm the new cluster, then write your keys into it

etcdctl member list -w table must show one member per configured location before you load anything. Write the exported keys with etcdctl put from your client, then point your clients at the new release’s address.

Remove the old release

cpln helm uninstall OLD_RELEASE_NAME --gvc OLD_GVC_NAME

Run it against the GVC the old release was installed into. Uninstalling removes the old release and the GVC it created, including its volume sets.

Scaling and Availability

Fault tolerance follows the location count, not the template. The cluster keeps serving while a majority of members is up: 3 locations tolerate the loss of 1, 5 tolerate 2, and 2 tolerate none (see Locations and Quorum). If you want automatic failover, use three locations.
Losing a minority of locations is survived automatically. The remaining members elect a new leader among themselves and keep committing writes; a member that comes back with its volume intact rejoins on its own.
Losing a majority stops writes but loses no committed data. The survivors hold the current keyspace and wait for quorum. Writes hang rather than failing fast, so clients need a command timeout; serializable reads (--consistency=s) keep succeeding against stale data. If the lost locations are not coming back, follow Recovering From a Lost Location.
A cpln helm upgrade restarts the members together — including an upgrade that changes nothing. Each member rejoins from its own volume and the keyspace is preserved, but expect a brief write outage while a leader is re-elected, so plan every upgrade as a short planned outage.
Change the location count only as a planned reprovision. Editing global.locations restarts every member with a new cluster list; the template performs no etcdctl member add or member remove. Change the GVC’s locations and the list together.
Do not run more than one member per location. global.locations[].replicas must be 1; a second member in a location adds cost and makes that location’s loss count double against quorum.

Recovering From a Lost Location

recovery.forceNewClusterInLocation is for the case where quorum is permanently gone — with two locations, that is the loss of either one. It is not needed for a location that is coming back: a member that returns with its volume intact rejoins on its own.

Confirm the loss is permanent

etcdctl member remove cannot help here, because removing a member itself requires quorum. If the location may return, wait for it instead.

Force a new single-member cluster

Set recovery.forceNewClusterInLocation to the surviving location’s name and upgrade the release:

cpln helm upgrade RELEASE_NAME oci://ghcr.io/controlplane-com/templates/etcd-multi-location \
  --version 2.0.0 --gvc GVC_NAME -f etcd-locations.yaml --set recovery.forceNewClusterInLocation=LOCATION

That member restarts as a single-member cluster rebuilt from its own write-ahead log and serves writes again.

Clear the setting

Set recovery.forceNewClusterInLocation back to "" and upgrade again. Leaving it set means the flag fires on every future restart of that member.

Reset the lost location's volume before it rejoins

A member from the lost location still holds the old cluster ID and refuses to start until its data directory is cleared. Uninstall and reinstall, or delete that volume, so the member bootstraps fresh.

Never set recovery.forceNewClusterInLocation to more than one location, and never leave it set. Two members both forcing a new cluster produce two divergent single-member clusters that cannot be merged.

The render-time check only compares this value against global.locations, not against the GVC. Naming a location the GVC does not have therefore passes validation, no member runs there, and the flag is never applied. Every surviving member logs a warning saying exactly that — see Troubleshooting if a recovery appears to do nothing.

If the Backend Quota Is Already Full

Every version of this template compacts, so a default install does not accumulate revisions in the first place. A backend can still reach the quota if the keyspace itself is genuinely large, or if autoCompactionRetention has been raised far enough that the retained history outgrows 2 GiB.

Turning compaction on, or tightening it, cannot rescue a cluster that has already filled its backend. Compaction stops further growth; it never shrinks an existing backend file. Once etcd has raised a NOSPACE alarm, writes stay rejected until an operator compacts, defragments each member and disarms the alarm — a cpln helm upgrade does none of that.

The symptom usually surfaces in the client rather than in etcd. A Patroni replica that cannot renew its lease exits cleanly, so it restart-loops with exitCode: 0 and reason: Completed and a climbing restart count — which reads as healthy and gets misdiagnosed as a database fault. Check etcd first. Both inspection commands are read-only and safe to run on a live cluster:

cpln workload exec RELEASE_NAME-etcd --gvc GVC_NAME --container etcd -- etcdctl endpoint status --cluster -w table
cpln workload exec RELEASE_NAME-etcd --gvc GVC_NAME --container etcd -- etcdctl alarm list

A DB SIZE close to 2.1 GB on every member, plus NOSPACE in the alarm list, confirms it. A healthy cluster prints nothing at all for alarm list. Recovery from there — compacting to a revision, defragmenting each member, then disarming the alarm — is an operator procedure this template deliberately does not perform, because each step is disruptive and the order matters. Follow etcd’s maintenance guide and plan it as a maintenance window.

Troubleshooting

When a member will not come up, its own [etcd] log lines name the reason:

cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-etcd"} |= "[etcd]"' --limit 50 --since 10m

Install fails: etcd-multi-location requires at least 2 locations in global.locations

Cause: The template validates global.locations at render time and rejects a list with fewer than two entries before any resource is created. This also happens when a --set global.locations[0].name=... override replaces the whole list with a single entry.Fix: Supply the full list in a values file (see Installation). For a single-location cluster use the etcd template.

Install or upgrade fails: the global.gvc values key was REMOVED

Cause: Your values still carry the 1.x global.gvc block. Version 2.0.0 no longer creates a GVC and refuses to render rather than let an in-place upgrade delete the one the old release created.Fix: Do not upgrade in place. Follow Migrating From Version 1: install 2.0.0 as a new release with global.locations, move the keyspace, then uninstall the old release.

Member exits with [etcd] FATAL: locations declared in global.locations are not in GVC

Symptom: A freshly installed member exits with code 1; its log names the missing location(s) and reads That is a bare majority: the cluster would form, report healthy, and have ZERO fault tolerance.Cause: global.locations names a location the GVC does not have. The members in the other locations would still form a cluster, but one with no fault tolerance, so a fresh member refuses to bootstrap it. An already-initialized member logs the same condition as a [etcd] WARNING and keeps serving.Fix: Add the missing location to the GVC, or remove it from global.locations, and redeploy. cpln gvc get GVC_NAME -o yaml shows the GVC’s list under spec.staticPlacement.locationLinks.

Member exits with [etcd] FATAL: quorum is unreachable

Cause: Fewer than a majority of the locations in global.locations exist in the GVC, so an election can never complete. A fresh member exits immediately rather than waiting forever; an already-initialized member logs a [etcd] WARNING and keeps running.Fix: Add the missing locations to the GVC or remove them from global.locations so that the GVC has a majority of the declared list — ideally all of it — and redeploy.

Member exits with [etcd] FATAL: this member is running in location ..., which is not in the configured location list

Cause: A replica started in a GVC location that global.locations does not list. Such a member can never appear in the cluster roster, so it exits rather than wait to join.Fix: Either add that location to global.locations and upgrade, or remove it from the GVC. Normally an unlisted GVC location runs nothing at all; this line only appears if a replica was placed there anyway.

Log shows [etcd] WARNING: could not read the location list of GVC

Cause: The member could not read its own GVC at boot — the API was briefly unavailable, or the RELEASE_NAME-etcd-gvc-policy policy is missing — so it skipped the location check and started anyway. The warning is informational; etcd runs normally.Fix: Nothing, if the policy exists and the warning was transient. If it recurs on every start, confirm the policy with cpln policy get RELEASE_NAME-etcd-gvc-policy -o yaml and that it grants view on the GVC the release is installed into.

etcdctl hangs, then reports context deadline exceeded

Cause: Fewer than a majority of members are reachable, so the cluster cannot commit writes; etcd waits for quorum rather than failing fast. Linearizable reads wait as well, while serializable reads (--consistency=s) keep answering from stale local data.Fix: Check which members are healthy and whether any replica is restarting:

cpln workload get-deployments RELEASE_NAME-etcd --gvc GVC_NAME -o yaml
cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-etcd"}' --limit 50 --since 10m

Once a majority is back, pending operations complete on their own. If the lost locations are gone for good, follow Recovering From a Lost Location. Always give clients a --command-timeout so an outage surfaces as an error instead of a hang.

endpoint status shows IS LEADER: true but writes still hang

Cause: IS LEADER is the member’s own view of itself. A member cut off from the others can keep reporting itself as leader for a short while after it has lost the ability to commit anything.Fix: Health-check with a linearizable read (etcdctl --command-timeout=5s get KEY, without --consistency=s), never with endpoint status alone. Then check etcdctl endpoint health --cluster and the deployment status as above.

Recovery appears to do nothing after setting recovery.forceNewClusterInLocation

Symptom: Every surviving member logs [etcd] WARNING: recovery.forceNewClusterInLocation is set to '...', which GVC '...' does not have.Cause: The value passed render-time validation because it is in global.locations, but the GVC has no such location, so no member runs there and --force-new-cluster is never applied.Fix: Set recovery.forceNewClusterInLocation to a location the GVC actually has — the one whose member survived — and upgrade again.

Writes fail with etcdserver: mvcc: database space exceeded

Cause: The backend quota is full and etcd has raised a cluster-wide NOSPACE alarm. Every member is read-only until the alarm is cleared.Fix: Confirm with etcdctl alarm list, then follow If the Backend Quota Is Already Full. Upgrading the template alone does not clear it.

Reads fail with etcdserver: mvcc: required revision has been compacted

Cause: A client asked for a revision older than the retention window — a get --rev or a watch resuming from a revision that auto-compaction has already discarded. With the default autoCompactionRetention: 1h only the last hour of history is readable.Fix: Read the current revision instead, or restart the watch from the current revision. If your application genuinely needs a longer history, raise tuning.autoCompactionRetention (for example 24h) and accept the larger backend.

Patroni replicas restart-loop with exitCode 0 and reason Completed

Cause: The Postgres cluster’s consensus store cannot accept writes — because this etcd cluster has lost quorum or its backend quota is full — so Patroni cannot renew its leader lease and exits cleanly. The restarts look healthy because the exit code is zero.Fix: Run the two read-only commands in If the Backend Quota Is Already Full and check etcdctl endpoint health --cluster before touching the database.

Important Notes

Do not cpln helm upgrade a 1.x release onto this version. It deletes the GVC the old release created and everything inside it. Follow Migrating From Version 1.
global.locations must match the GVC’s locations. A declared location the GVC lacks makes a fresh member refuse to start with a named [etcd] FATAL line; a GVC location you did not list runs nothing. Read the GVC’s list with cpln gvc get GVC_NAME -o yaml before installing.
Quorum arithmetic decides your failure tolerance. Two locations survive zero losses. If you want automatic failover, use three.
Every cpln helm upgrade restarts the members together, including a no-op one. Expect a brief write outage; the data survives. Plan upgrades as a short planned outage.
Changing global.locations reprovisions the cluster. Plan it as a maintenance window and change the GVC’s locations together with the list.
Under quorum loss, writes hang rather than failing fast, and serializable reads keep succeeding against stale data. Give clients a short command timeout and health-check with a linearizable read, never with IS LEADER alone.
No TLS, no authentication. Scope internalAccess with workload-list whenever the GVC holds workloads that should not read or write the keyspace; list only your clients.
Compaction is required and cannot be switched off. A full backend quota puts the cluster into a read-only NOSPACE alarm that an upgrade cannot clear.
recovery.forceNewClusterInLocation is for a permanent loss of quorum only. Never set it to more than one location, and clear it as soon as the cluster is back.
Uninstalling deletes the volume sets, and with them the entire keyspace. The volume set takes a final snapshot first.

External References

etcd Documentation

Official etcd documentation

Clustering Guide

Static bootstrapping and cluster membership

Tuning for Latency

Heartbeat and election timeout guidance for cross-region clusters

Maintenance and Compaction

Compaction, defragmentation, and clearing a NOSPACE alarm

Disaster Recovery

Snapshotting and restoring a cluster with etcdctl and etcdutl