etcd Multi-Location

Version 2.0.0 no longer creates a GVC — it deploys into one you already have, and there is no in-place upgrade path from 1.x. A helm upgrade across that boundary deletes the GVC the old release created and every workload, volume set and identity inside it. See Migrating From Version 1.

Overview

etcd is a strongly consistent key-value store used as the coordination layer for leader election, distributed locking, and service configuration. This template deploys a single stretched etcd cluster with exactly one member per Control Plane location, sharing one raft quorum across regions, with raft timers tuned for cross-region round trips and auto-compaction enabled. It is also the consensus store used by the postgres-multi-location template, and is independently useful as a cross-region coordination store for your own services.

This template deploys into a GVC you already have, and that GVC must have at least 2 locations. It does not create, provision or manage a GVC. For a single-location cluster, use the etcd template instead.

What Gets Created

Stateful etcd Workload — (RELEASE_NAME-etcd): one member in each configured location, using replicaDirect addressing so every member is individually reachable. Client API on 2379, raft peer traffic on 2380.
Volume Set — (RELEASE_NAME-etcd-vs): persistent storage per member at /var/lib/etcd for the raft write-ahead log and snapshots. ext4, general-purpose-ssd, final snapshot on delete, 7-day snapshot retention.
Secret — (RELEASE_NAME-etcd-startup): an opaque startup script that computes each member’s name, peer URL, and the full cluster list at container start from the location it is running in.
Identity & Two Policies — An identity bound to the workload, granted reveal on the startup script secret and nothing else, plus view on exactly the one GVC this release installs into. The second grant is what lets each member read its own GVC’s location list at boot; it is scoped to that single GVC, never to the whole org.

No GVC resource is created. Every resource above lands in the GVC you install into, and the members read that GVC’s name from the platform at runtime rather than from Helm, so a hostname can never drift from where the workload actually runs. Members find each other over per-replica internal DNS (replica-0.RELEASE_NAME-etcd.LOCATION.GVC_NAME.cpln.local:2380). There is no operator, no discovery service, and no join step — every member receives the full cluster list up front and they elect a leader among themselves.

Installation

The only prerequisite is an existing GVC whose locations include every entry in global.locations — read its list with cpln gvc get GVC_NAME -o yaml and compare spec.staticPlacement.locationLinks before you install. Nothing else is needed: no cloud account, no bucket, no pre-created secret, no custom domain. To install, follow the instructions for your preferred method:

UI

Browse, install, and manage templates visually

CLI

Manage templates from your terminal

Terraform

Declare templates in your Terraform configurations

Pulumi

Declare templates in your Pulumi programs

Configuration

The default values.yaml for this template:

# ─── Locations ────────────────────────────────────────────────────────────────
# This chart deploys into the GVC you install into — it does NOT create one.
# Every location listed here MUST already exist in that GVC: a location the GVC
# lacks contributes a declared etcd member that can never start, which shrinks
# the cluster to a bare majority while every status surface reads healthy.
# The members refuse to bootstrap in that state rather than form it.
#
# Lives under `global` so a parent chart (postgres-multi-location consumes this
# one as a subchart) sets the list once and Helm propagates it here — the two
# lists can then never be edited apart.

global:
  # One etcd member per location. Minimum 2; 3 survives losing one location,
  # 5 survives losing two. See the quorum table in the README — 3 is the
  # recommended production shape. This default is the smallest cluster the
  # chart allows, because no location list can be right for every GVC: set it
  # to YOUR GVC's locations before installing.
  locations:
    - name: aws-us-east-1
    - name: aws-us-west-2

# ─── etcd ─────────────────────────────────────────────────────────────────────
image: controlplanecorporation/etcd:0.1

resources:
  cpu: 500m
  memory: 512Mi

# Raft timers, tuned for cross-region round trips (measured 63-140 ms).
# Raise both if your locations are more than ~250 ms apart.
tuning:
  heartbeatIntervalMs: 250 # ~0.5-1.5x the worst round trip between locations
  electionTimeoutMs: 5000 # must be >= 10x heartbeatIntervalMs; maximum 50000
  # etcd keeps every superseded revision until told otherwise, so compaction is
  # required rather than optional and cannot be switched off here.
  autoCompactionMode: periodic # periodic (retention is a duration) or revision (a revision count)
  autoCompactionRetention: 1h # periodic needs an explicit unit (1h, 30m, 24h); revision takes a count
  quotaBackendBytes: 0 # backend size limit in bytes; 0 = etcd's own default of 2 GiB

volumeset:
  capacity: 10 # initial capacity in GiB (minimum is 10)

internalAccess:
  type: same-gvc # options: same-gvc, same-org, workload-list
  workloads: [] # only used when type is workload-list; the etcd workload is added automatically
  #- //gvc/GVC_NAME/workload/WORKLOAD_NAME

# ─── Disaster recovery ────────────────────────────────────────────────────────
recovery:
  # EMERGENCY ONLY. Set to the surviving location's name to restart that member
  # as a new single-member cluster after permanently losing quorum. Read the
  # "Recovering from a lost location" section of the README first.
  forceNewClusterInLocation: ""

Locations and Quorum

global.locations is the member map: one etcd member is deployed in each listed location, and at least 2 locations are required. Every name in it must already be one of the locations of the GVC you install into. The default list is the smallest cluster the chart allows rather than a recommendation — no built-in list can match your GVC, so set it before installing. The list lives under global so a parent chart — postgres-multi-location consumes this template as a subchart — sets it once and Helm propagates it here. The two lists can then never be edited apart. Each location runs exactly one member; replicas is fixed at 1 and any other value fails a standalone render. etcd commits a write only when a majority of members accept it. Because there is exactly one member per location, the location is the failure domain:

Locations	Majority needed	Location losses survived	What that means
2	2	0	Losing either one stops writes. The survivor holds current data, but recovery is manual.
3	2	1	Automatic failover. The recommended shape.
4	3	1	No better than 3, and costs more.
5	3	2	Survives losing two locations.

With N locations you survive floor((N-1)/2) losses, so an even count never buys anything over the odd count below it. Two locations is permitted as a deliberate warm-standby topology, but it survives nothing automatically.

Changing global.locations reprovisions the cluster: every member restarts with a new cluster list. This is not etcd’s graceful member add / member remove path, so plan it as a maintenance window.

Matching the Location List to the GVC

The platform does not validate the pairing in either direction, and both mismatches are dangerous in different ways. The template closes both.

`global.locations`	The GVC’s locations	What happens
3	The same 3	Correct. Survives losing one location.
3	2 of those 3	Would otherwise form and report healthy with zero fault tolerance — a `new` bootstrap needs only a majority of the declared set. The template refuses to bootstrap instead.
3	1 of those 3	Quorum is arithmetically unreachable. The template refuses to bootstrap rather than wait forever for an election that cannot complete.
3	Those 3 plus another	The extra location runs nothing — the workload’s default `minScale`/`maxScale` are `0`, and only the declared locations get a real replica count. Its deployment reads `This workload location is deactivated because maxScale is set to 0`.

Each member checks its own GVC at boot, using the scoped view grant described in What Gets Created, and the check is deliberately asymmetric:

On a fresh data directory, any declared location the GVC lacks is a hard failure. The container exits with code 1 and logs a [etcd] FATAL: line naming the missing location and how many declared members could actually start.
On an already-initialized member it is a warning only, and the member keeps serving. A location removed from a GVC is indistinguishable from a location that is down, so failing here would turn “lost one member” into “lost the cluster”.
If the GVC read itself fails — a control-plane blip, or a missing policy — the check is skipped with a warning. It never blocks a start on its own unavailability.

Look for [etcd] lines in cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-etcd"}' when a member will not come up.

Image and Resources

image — The etcd image to run (etcd 3.6.5).
resources.cpu / resources.memory — CPU and memory allocated to each member (default: 500m, 512Mi).

Raft Tuning

tuning.heartbeatIntervalMs — Leader heartbeat interval, roughly 0.5–1.5× the worst round trip between your locations.
tuning.electionTimeoutMs — How long a follower waits before campaigning. Must be at least 10× the heartbeat interval, and at most 50000. Both bounds are enforced at render time.

The defaults detect a dead leader in about 5 seconds across an AWS us-east ↔ eu-central ↔ us-west triangle. Locations further apart (US ↔ Asia-Pacific is 350–400 ms) need both values raised in proportion.

Storage

volumeset.capacity — Persistent volume size in GiB per member for the raft write-ahead log and snapshots (minimum 10).

Compaction and Backend Growth

etcd never discards a superseded revision on its own. Every write creates a new revision and the old one is kept until a compaction removes it, so the backend grows with time alone whenever a client writes on a timer — Patroni, for example, renews its leader lease roughly every 10 seconds, which produces new revisions whether or not any application data changes. Measured on an otherwise idle cluster: about 151,000 revisions and 19 MB per day, which reaches etcd’s default 2 GiB backend quota in roughly 110 days. When the quota is reached, etcd raises a cluster-wide NOSPACE alarm and every member goes read-only until an operator intervenes. Auto-compaction is what prevents that. It has always been enabled in this template; since version 1.0.2 it is also configurable, alongside the backend quota:

tuning:
  autoCompactionMode: periodic # periodic (retention is a duration) or revision (a revision count)
  autoCompactionRetention: 1h # periodic needs an explicit unit (1h, 30m, 24h); revision takes a count
  quotaBackendBytes: 0 # backend size limit in bytes; 0 = etcd's own default of 2 GiB

tuning.autoCompactionMode — How autoCompactionRetention is read. periodic treats it as a time window; revision treats it as a number of revisions to keep. There is no “off” — an unrecognized mode is rejected at render time.
tuning.autoCompactionRetention — How much history is kept before old revisions are discarded. The default 1h keeps the backend flat and is the value this template is tested at. Raising it to 24h or more buys a longer history window at the cost of a larger backend.
tuning.quotaBackendBytes — Backend size ceiling in bytes. 0 (the default) means the flag is not passed at all and etcd applies its own 2 GiB limit. Leave it at 0 unless the keyspace genuinely outgrows that; 8 GiB (8589934592) is etcd’s own suggested maximum, above which it warns at startup.

In periodic mode a retention value without a unit means hours. etcd reads a bare 30 as thirty hours, not thirty minutes. This template rejects an unsuffixed value at render time rather than letting it become a silently wrong window — write 30m, 1h or 24h.

quotaBackendBytes takes a plain byte count, not a size suffix. etcd itself refuses --quota-backend-bytes 2Gi at boot and the cluster would crash-loop, so a 2Gi-style value is rejected at render time instead. Write 2147483648.

Compaction is deliberately not switchable off: a retention of 0, an unrecognized mode and a negative quota (which etcd reads as “no quota at all”) are each rejected at render time. Each one produces a chart that installs cleanly and fails weeks later.

Compaction frees pages for reuse inside the backend file; it does not shrink the file. The reported dbSize therefore plateaus rather than dropping, while dbSizeInUse falls back to the size of the live keyspace. That is the expected behavior and is sufficient to stay under the quota, because reclaimed pages are reused for new writes. Only etcdctl defrag returns space to the filesystem, and this template does not automate it — defragmentation blocks the member it runs on.Testing measured the mechanism directly on a shortened retention window: the compacted revision advanced from 1 to 4001, dbSizeInUse collapsed from 5.76 MB to 20,480 bytes while dbSize stayed at about 5.8 MB, and reads of pre-compaction revisions then failed with required revision has been compacted. On this template’s own three-location run at the default 1h, compaction fired at the one-hour mark, freed 569 KB, and dbSize stayed flat afterwards under load. The multi-day dbSize plateau is derived from that mechanism rather than observed over days.

etcdctl endpoint status reports QUOTA | 0 B when quotaBackendBytes is 0, because that column reflects the flag rather than the effective limit. The effective quota in that case is still etcd’s 2 GiB default, which etcd logs at startup as enabled backend quota with default value.

Internal Access

The internalAccess section controls which workloads can reach the client API on 2379. Cross-location traffic inside one GVC is same-GVC traffic, so the default covers a stretched cluster with no extra rule.

Type	Description
`same-gvc`	Allow access from all workloads in the same GVC (recommended)
`same-org`	Allow access from all workloads in the same organization
`workload-list`	Allow access only from the specific workloads listed in `internalAccess.workloads`

With workload-list, list only your clients — never the etcd workload itself. The internal firewall governs all inbound internal traffic, including the raft peer traffic between the members of this same workload, so a list naming only clients used to cut the cluster off from itself: on an earlier version, every member was dead within about two minutes of the upgrade while Helm reported upgraded successfully. The template now appends its own workload link for you, so scoping access is safe — a client that is not listed is still refused, and the cluster stays whole.

There is deliberately no public access: etcd in this template runs without TLS and without authentication. Anything permitted by internalAccess has full read/write access to the entire keyspace. Use workload-list if the GVC contains workloads that should not have it.

A firewall change takes roughly 30 seconds to a few minutes to take effect — over two minutes was measured on this template, and other templates have taken longer still. Re-test after waiting rather than concluding the setting was ignored.

Connecting to etcd

Target	Address
Client API, load-balanced across locations	`RELEASE_NAME-etcd.GVC_NAME.cpln.local:2379`
A specific location’s member	`replica-0.RELEASE_NAME-etcd.LOCATION.GVC_NAME.cpln.local:2379`
Raft peer traffic (members only)	Port `2380`

GVC_NAME is the GVC you installed the release into — the template creates none of its own. Point clients at one endpoint per location so they can fail over. From inside the cluster:

cpln workload exec RELEASE_NAME-etcd --gvc GVC_NAME --container etcd -- etcdctl member list -w table
cpln workload exec RELEASE_NAME-etcd --gvc GVC_NAME --container etcd -- etcdctl endpoint status --cluster -w table
cpln workload exec RELEASE_NAME-etcd --gvc GVC_NAME --container etcd -- etcdctl put /demo/key hello
cpln workload exec RELEASE_NAME-etcd --gvc GVC_NAME --container etcd -- etcdctl --command-timeout=5s get /demo/key

Every member is named RELEASE_NAME-etcd-LOCATION, so member list maps one-to-one onto your location list.

Allow about two minutes of convergence after a cold install before concluding that a member is unreachable — cross-region service discovery can lag a workload reporting ready by well over a minute.

Ports

Port	Protocol	Description
`2379`	TCP	etcd client API (internal only)
`2380`	TCP	Raft peer traffic between members

Failover Behavior

Measured on a three-location cluster (aws-us-east-1, aws-eu-central-1, aws-us-west-2) with the default 5 s election timeout:

Event	Result
Idle stability	10.8 hours with zero spontaneous elections
Leader crash	New leader elected in 5.59 s, with 0 failed writes on a client pinned to a surviving follower
Planned replica stop	Leadership handed over in 502 ms, 0 failed writes across 570 samples

Write latency is bounded by one cross-region round trip, because the leader needs a follower acknowledgement before it can commit. Measured with the leader in aws-eu-central-1:

Client location	Write latency (p50)
`aws-eu-central-1` (co-located with the leader)	95.7 ms
`aws-us-east-1`	185.7 ms
`aws-us-west-2` (furthest from the leader)	236.4 ms

A client co-located with the leader does not get local-write latency — one cross-region round trip is the floor for any stretched quorum.

Upgrades

Every helm upgrade restarts all members at roughly the same time — a rollout of about 105 seconds, with 20–30 seconds of lost quorum inside it. Measured on three locations and reproduced at 104 seconds on a second upgrade. Writes time out while quorum is gone, the cluster recovers on its own, and the data survives. A no-op upgrade that changes nothing costs exactly the same, so there is no cheaper kind of upgrade to plan around — treat every change, including one that only touches a firewall rule, as a short planned outage.

Nothing serializes the restart: the field that would limit it (rolloutOptions.maxUnavailableReplicas) is not retained by the platform, so the chart deliberately does not set it. Running more members per location has not been tested and should not be assumed to help.

Migrating From Version 1

Versions before 2.0.0 created their own GVC. Do not helm upgrade a 1.x release onto 2.0.0. Once the chart stops declaring a GVC, Helm prunes the one the old release created — and deleting a GVC deletes every workload, volume set and identity inside it. Measured on a sibling template: everything was gone in about six seconds, while the command printed upgraded successfully.

The chart refuses to render when your values still carry the 1.x global.gvc key, so a values-carrying upgrade fails safely before any resource is touched. That guard cannot fire on an upgrade run with no values file at all, because it then sees only 2.0.0’s own defaults. The procedure below is the safety; the render guard is only a backstop.

Install 2.0.0 as a new release against an existing GVC, move the keyspace across, then remove the old release.

Choose the target GVC

Pick a GVC whose locations are exactly the ones you want members in, and read them back with cpln gvc get GVC_NAME -o yaml — spec.staticPlacement.locationLinks is the list. Do not point the new release at the GVC the old release created: that GVC is still owned by the old Helm release, and uninstalling it later would take the new workloads with it.

Snapshot the old cluster's keyspace

cpln workload exec OLD_RELEASE-etcd --gvc OLD_GVC_NAME --container etcd \
  -- etcdctl get "" --prefix -w json > keyspace.json

Install 2.0.0 as a new release

Replace global.gvc.name and global.gvc.locations in your old values with a single top-level global.locations list naming the chosen GVC’s locations, then install into that GVC.

Confirm the new cluster, then write your keys into it

etcdctl member list -w table must show one member per configured location before you load anything.

Remove the old release

Uninstalling the old release removes it and the GVC it created, including its volume sets. Run it against the GVC the old release was installed into.

Behavior Under Quorum Loss

When a majority of members is unreachable, the cluster stops committing writes. What that looks like from a client is easy to misread:

Writes hang rather than failing fast. They block until they time out, so always give clients a short --command-timeout (or the client-library equivalent) — otherwise connections pile up against a cluster that cannot commit.
Serializable reads keep succeeding against stale data. A read issued with --consistency=s is served from the local member’s own store and never notices the loss of quorum.
IS LEADER: true is not proof of leadership. An isolated survivor keeps reporting itself as leader for about 6 seconds while unable to commit anything. Health-check with a linearizable read (etcdctl get KEY, without --consistency=s), never with endpoint status.

Never suspend a location for this workload. Suspending and resuming a location permanently withdraws that location’s endpoints from the other locations’ service discovery while every status surface still reports healthy. Add or remove locations by editing the GVC and global.locations together instead.

Recovering from a Lost Location

recovery.forceNewClusterInLocation is for the case where quorum is permanently gone — with two locations, that is the loss of either one. It is not needed for a location that is coming back: a member that returns with its volume intact rejoins on its own in well under a minute.

Confirm the loss is permanent

etcdctl member remove cannot help here, because removing a member itself requires quorum.

Force a new single-member cluster

Set recovery.forceNewClusterInLocation to the surviving location’s name and upgrade the release. That member restarts as a single-member cluster rebuilt from its own write-ahead log and serves writes again immediately.

Clear the setting

Set recovery.forceNewClusterInLocation back to "" and upgrade again. Leaving it set means the flag fires on every future restart of that member.

Reset the lost location's volume before it rejoins

The evicted members still hold the old cluster ID and refuse to start until their data directory is cleared. Uninstall and reinstall, or delete that volume, so the member bootstraps fresh.

Never set recovery.forceNewClusterInLocation to more than one location, and never leave it set. Two members both forcing a new cluster produce two divergent single-member clusters that cannot be merged.

The render-time check only compares this value against global.locations, not against the GVC. Naming a location the GVC does not have therefore passes validation, no member runs there, and the flag is silently never applied. Every surviving member logs a warning saying exactly that — [etcd] WARNING: recovery.forceNewClusterInLocation is set to '...', which GVC '...' does not have — so read the logs if a recovery appears to do nothing.

If the Backend Quota Fills

Every version of this template compacts, so a default install does not accumulate revisions in the first place. A backend can still reach the quota if the keyspace itself is genuinely large, or if autoCompactionRetention has been raised far enough that the retained history outgrows 2 GiB.

Turning compaction on, or tightening it, cannot rescue a cluster that has already filled its backend. Compaction stops further growth; it never shrinks an existing backend file. Once etcd has raised a NOSPACE alarm, writes stay rejected until an operator compacts, defragments each member and disarms the alarm — a helm upgrade does none of that.

The symptom usually surfaces in the client rather than in etcd. A Patroni replica that cannot renew its lease exits cleanly, so it restart-loops with exitCode: 0 and reason: Completed and a climbing restart count — which reads as healthy and gets misdiagnosed as a database fault. Check etcd first. Both inspection commands are read-only and safe to run on a live cluster:

cpln workload exec RELEASE_NAME-etcd --gvc GVC_NAME --container etcd -- etcdctl endpoint status --cluster -w table
cpln workload exec RELEASE_NAME-etcd --gvc GVC_NAME --container etcd -- etcdctl alarm list

Add --location LOCATION to target a specific location’s member; without it the CLI picks one for you and names it in its output. endpoint status --cluster reports every member from whichever one you reach. A DB SIZE close to 2.1 GB on every member, plus NOSPACE in the alarm list, confirms it. A healthy cluster prints nothing at all for alarm list. Recovery from there — compacting to a revision, defragmenting each member, then disarming the alarm — is an operator procedure this template deliberately does not perform, because each step is disruptive and the order matters. Follow etcd’s maintenance guide and plan it as a maintenance window.

External References

etcd Documentation

Official etcd documentation

Maintenance and Compaction

Compaction, defragmentation, and clearing a NOSPACE alarm

Tuning for Latency

Heartbeat and election timeout guidance for cross-region clusters

Disaster Recovery

Recovering an etcd cluster that has lost quorum