etcd Multi-Location

Overview

etcd is a strongly consistent key-value store used as the coordination layer for leader election, distributed locking, and service configuration. This template deploys a single stretched etcd cluster with exactly one member per Control Plane location, sharing one raft quorum across regions, with raft timers tuned for cross-region round trips and auto-compaction enabled. It is also the consensus store used by the postgres-multi-location template, and is independently useful as a cross-region coordination store for your own services.

This template creates a new GVC and requires at least 2 locations. For a single-location cluster, use the etcd template instead.

What Gets Created

GVC — A new GVC pinned to the configured locations via static placement. The GVC is what pins the deployment’s locations, so it is always created.
Stateful etcd Workload — (RELEASE_NAME-etcd): one member in each configured location, using replicaDirect addressing so every member is individually reachable. Client API on 2379, raft peer traffic on 2380.
Volume Set — (RELEASE_NAME-etcd-vs): persistent storage per member at /var/lib/etcd for the raft write-ahead log and snapshots. ext4, general-purpose-ssd, final snapshot on delete, 7-day snapshot retention.
Secret — (RELEASE_NAME-etcd-startup): an opaque startup script that computes each member’s name, peer URL, and the full cluster list at container start from the location it is running in.
Identity & Policy — An identity bound to the workload with reveal access to the startup script secret and nothing else.

Members find each other over per-replica internal DNS (replica-0.RELEASE_NAME-etcd.LOCATION.GVC_NAME.cpln.local:2380). There is no operator, no discovery service, and no join step — every member receives the full cluster list up front and they elect a leader among themselves.

Installation

This template has no external prerequisites. To install, follow the instructions for your preferred method:

UI

Browse, install, and manage templates visually

CLI

Manage templates from your terminal

Terraform

Declare templates in your Terraform configurations

Pulumi

Declare templates in your Pulumi programs

Configuration

The default values.yaml for this template:

# ─── GVC and locations ────────────────────────────────────────────────────────
# Lives under `global` so a parent chart (postgres-multi-location) sets it once
# and Helm propagates it here. See README before changing.

global:
  gvc:
    name: etcd-multi-location-gvc # the GVC this chart creates
    # One etcd member per location. Minimum 2; 3 survives losing one location,
    # 5 survives losing two. See the quorum table in the README.
    locations:
      - name: aws-us-east-1
      - name: aws-eu-central-1
      - name: aws-us-west-2

# ─── etcd ─────────────────────────────────────────────────────────────────────
image: controlplanecorporation/etcd:0.1

resources:
  cpu: 500m
  memory: 512Mi

# Raft timers, tuned for cross-region round trips (measured 63-140 ms).
# Raise both if your locations are more than ~250 ms apart.
tuning:
  heartbeatIntervalMs: 250 # ~0.5-1.5x the worst round trip between locations
  electionTimeoutMs: 5000 # must be >= 10x heartbeatIntervalMs; maximum 50000

volumeset:
  capacity: 10 # initial capacity in GiB (minimum is 10)

internalAccess:
  type: same-gvc # options: same-gvc, same-org, workload-list
  workloads: [] # only used when type is workload-list
  #- //gvc/GVC_NAME/workload/WORKLOAD_NAME

# ─── Disaster recovery ────────────────────────────────────────────────────────
recovery:
  # EMERGENCY ONLY. Set to the surviving location's name to restart that member
  # as a new single-member cluster after permanently losing quorum. Read the
  # "Recovering from a lost location" section of the README first.
  forceNewClusterInLocation: ""

Locations and Quorum

global.gvc.name — Name of the GVC this template creates. It must not name a GVC that already exists.
global.gvc.locations — The member map: one etcd member is deployed in each listed location. At least 2 locations are required.

Helm owns the GVC this template creates. If global.gvc.name matches a GVC that already exists, Helm adopts it and a later uninstall deletes that GVC along with everything inside it. Always pick a name no other release uses.

etcd commits a write only when a majority of members accept it. Because there is exactly one member per location, the location is the failure domain:

Locations	Majority needed	Location losses survived	What that means
2	2	0	Losing either one stops writes. The survivor holds current data, but recovery is manual.
3	2	1	Automatic failover. The recommended shape.
4	3	1	No better than 3, and costs more.
5	3	2	Survives losing two locations.

With N locations you survive floor((N-1)/2) losses, so an even count never buys anything over the odd count below it. Two locations is permitted as a deliberate warm-standby topology, but it survives nothing automatically. The global.gvc block lives under global so that a parent chart — postgres-multi-location consumes this template as a subchart — sets the GVC and location list once and Helm propagates it. Each location runs exactly one member; any other replica count fails at render time.

Changing global.gvc.locations reprovisions the cluster: every member restarts with a new cluster list. This is not etcd’s graceful member add / member remove path, so plan it as a maintenance window.

Image and Resources

image — The etcd image to run (etcd 3.6.5).
resources.cpu / resources.memory — CPU and memory allocated to each member (default: 500m, 512Mi).

Raft Tuning

tuning.heartbeatIntervalMs — Leader heartbeat interval, roughly 0.5–1.5× the worst round trip between your locations.
tuning.electionTimeoutMs — How long a follower waits before campaigning. Must be at least 10× the heartbeat interval, and at most 50000. Both bounds are enforced at render time.

The defaults detect a dead leader in about 5 seconds across an AWS us-east ↔ eu-central ↔ us-west triangle. Locations further apart (US ↔ Asia-Pacific is 350–400 ms) need both values raised in proportion.

Storage

volumeset.capacity — Persistent volume size in GiB per member for the raft write-ahead log and snapshots (minimum 10).

Auto-compaction is enabled by design (periodic, every hour) and is not configurable. The first compaction happens an hour after a member starts, and a restart resets that clock. Without it, a continuously written cluster grows revisions until it reaches etcd’s 2 GiB backend quota and goes read-only. Compaction reclaims pages for reuse rather than shrinking the file, so the reported dbSize plateaus instead of dropping — only etcdctl defrag returns space to the filesystem.

Internal Access

The internalAccess section controls which workloads can reach the client API on 2379. Cross-location traffic inside one GVC is same-GVC traffic, so the default covers a stretched cluster with no extra rule.

Type	Description
`same-gvc`	Allow access from all workloads in the same GVC (recommended)
`same-org`	Allow access from all workloads in the same organization
`workload-list`	Allow access only from the specific workloads listed in `internalAccess.workloads`

There is deliberately no public access: etcd in this template runs without TLS and without authentication. Anything permitted by internalAccess has full read/write access to the entire keyspace. Use workload-list if the GVC contains workloads that should not have it.

Firewall changes take up to roughly two and a half minutes to take effect. Re-test after waiting rather than concluding the setting was ignored.

Connecting to etcd

Target	Address
Client API, load-balanced across locations	`RELEASE_NAME-etcd.GVC_NAME.cpln.local:2379`
A specific location’s member	`replica-0.RELEASE_NAME-etcd.LOCATION.GVC_NAME.cpln.local:2379`
Raft peer traffic (members only)	Port `2380`

Point clients at one endpoint per location so they can fail over. From inside the cluster:

cpln workload exec RELEASE_NAME-etcd --gvc GVC_NAME --container etcd -- etcdctl member list -w table
cpln workload exec RELEASE_NAME-etcd --gvc GVC_NAME --container etcd -- etcdctl endpoint status --cluster -w table
cpln workload exec RELEASE_NAME-etcd --gvc GVC_NAME --container etcd -- etcdctl put /demo/key hello
cpln workload exec RELEASE_NAME-etcd --gvc GVC_NAME --container etcd -- etcdctl --command-timeout=5s get /demo/key

Every member is named RELEASE_NAME-etcd-LOCATION, so member list maps one-to-one onto your location list.

Allow about two minutes of convergence after a cold install before concluding that a member is unreachable — cross-region service discovery can lag a workload reporting ready by well over a minute.

Ports

Port	Protocol	Description
`2379`	TCP	etcd client API (internal only)
`2380`	TCP	Raft peer traffic between members

Failover Behavior

Measured on a three-location cluster (aws-us-east-1, aws-eu-central-1, aws-us-west-2) with the default 5 s election timeout:

Event	Result
Idle stability	10.8 hours with zero spontaneous elections
Leader crash	New leader elected in 5.59 s, with 0 failed writes on a client pinned to a surviving follower
Planned replica stop	Leadership handed over in 502 ms, 0 failed writes across 570 samples

Write latency is bounded by one cross-region round trip, because the leader needs a follower acknowledgement before it can commit. Measured with the leader in aws-eu-central-1:

Client location	Write latency (p50)
`aws-eu-central-1` (co-located with the leader)	95.7 ms
`aws-us-east-1`	185.7 ms
`aws-us-west-2` (furthest from the leader)	236.4 ms

A client co-located with the leader does not get local-write latency — one cross-region round trip is the floor for any stretched quorum.

Upgrades

A helm upgrade takes the whole cluster down for about 66 seconds (measured on three locations). Members in every location restart together, quorum is lost, and writes time out until it returns. The cluster recovers on its own, but every configuration change — including one that only changes a firewall rule — costs that window, so plan upgrades as a short planned outage.

Nothing serializes the restart: the field that would limit it (rolloutOptions.maxUnavailableReplicas) is not retained by the platform, so the chart deliberately does not set it. Running more members per location has not been tested and should not be assumed to help.

Behavior Under Quorum Loss

When a majority of members is unreachable, the cluster stops committing writes. What that looks like from a client is easy to misread:

Writes hang rather than failing fast. They block until they time out, so always give clients a short --command-timeout (or the client-library equivalent) — otherwise connections pile up against a cluster that cannot commit.
Serializable reads keep succeeding against stale data. A read issued with --consistency=s is served from the local member’s own store and never notices the loss of quorum.
IS LEADER: true is not proof of leadership. An isolated survivor keeps reporting itself as leader for about 6 seconds while unable to commit anything. Health-check with a linearizable read (etcdctl get KEY, without --consistency=s), never with endpoint status.

Never suspend a location for this workload. Suspending and resuming a location permanently withdraws that location’s endpoints from the other locations’ service discovery while every status surface still reports healthy. Add or remove locations by editing global.gvc.locations instead.

Recovering from a Lost Location

recovery.forceNewClusterInLocation is for the case where quorum is permanently gone — with two locations, that is the loss of either one. It is not needed for a location that is coming back: a member that returns with its volume intact rejoins on its own in well under a minute.

Confirm the loss is permanent

etcdctl member remove cannot help here, because removing a member itself requires quorum.

Force a new single-member cluster

Set recovery.forceNewClusterInLocation to the surviving location’s name and upgrade the release. That member restarts as a single-member cluster rebuilt from its own write-ahead log and serves writes again immediately.

Clear the setting

Set recovery.forceNewClusterInLocation back to "" and upgrade again. Leaving it set means the flag fires on every future restart of that member.

Reset the lost location's volume before it rejoins

The evicted members still hold the old cluster ID and refuse to start until their data directory is cleared. Uninstall and reinstall, or delete that volume, so the member bootstraps fresh.

Never set recovery.forceNewClusterInLocation to more than one location, and never leave it set. Two members both forcing a new cluster produce two divergent single-member clusters that cannot be merged.

External References

etcd Documentation

Official etcd documentation

Tuning for Latency

Heartbeat and election timeout guidance for cross-region clusters

Disaster Recovery

Recovering an etcd cluster that has lost quorum