> ## Documentation Index
> Fetch the complete documentation index at: https://docs.controlplane.com/llms.txt
> Use this file to discover all available pages before exploring further.

# etcd Multi-Location

> Deploy etcd Multi-Location on Control Plane using the Template Catalog. Covers configuration, volumes, quorum sizing, and a stretched key-value cluster with exactly one member per location.

## Overview

etcd is a strongly consistent key-value store used as the coordination layer for leader election, distributed locking, and service configuration. This template deploys a **single stretched etcd cluster with exactly one member per Control Plane location**, sharing one raft quorum across regions, with raft timers tuned for cross-region round trips and auto-compaction enabled.

It is also the consensus store used by the [`postgres-multi-location`](/template-catalog/templates/postgres-multi-location) template, and is independently useful as a cross-region coordination store for your own services.

<Note>
  This template creates a new GVC and requires at least 2 locations. For a single-location cluster, use the [etcd](/template-catalog/templates/etcd) template instead.
</Note>

### What Gets Created

* **GVC** — A new GVC pinned to the configured locations via static placement. The GVC is what pins the deployment's locations, so it is always created.
* **Stateful etcd Workload** — (`RELEASE_NAME-etcd`): one member in each configured location, using `replicaDirect` addressing so every member is individually reachable. Client API on `2379`, raft peer traffic on `2380`.
* **Volume Set** — (`RELEASE_NAME-etcd-vs`): persistent storage per member at `/var/lib/etcd` for the raft write-ahead log and snapshots. ext4, general-purpose-ssd, final snapshot on delete, 7-day snapshot retention.
* **Secret** — (`RELEASE_NAME-etcd-startup`): an opaque startup script that computes each member's name, peer URL, and the full cluster list at container start from the location it is running in.
* **Identity & Policy** — An identity bound to the workload with `reveal` access to the startup script secret and nothing else.

Members find each other over per-replica internal DNS (`replica-0.RELEASE_NAME-etcd.LOCATION.GVC_NAME.cpln.local:2380`). There is no operator, no discovery service, and no join step — every member receives the full cluster list up front and they elect a leader among themselves.

## Installation

This template has no external prerequisites. To install, follow the instructions for your preferred method:

<CardGroup cols={2}>
  <Card title="UI" href="/template-catalog/install-manage/ui" icon="laptop">
    Browse, install, and manage templates visually
  </Card>

  <Card title="CLI" href="/template-catalog/install-manage/cli" icon="terminal">
    Manage templates from your terminal
  </Card>

  <Card title="Terraform" href="/template-catalog/install-manage/terraform" icon={<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 128 128"><g fill-rule="evenodd"><path d="M77.941 44.5v36.836L46.324 62.918V26.082zm0 0" fill="#5c4ee5"/><path d="M81.41 81.336l31.633-18.418V26.082L81.41 44.5zm0 0" fill="#4040b2"/><path d="M11.242 42.36L42.86 60.776V23.941L11.242 5.523zm0 0M77.941 85.375L46.324 66.957v36.82l31.617 18.418zm0 0" fill="#5c4ee5"/></g></svg>}>
    Declare templates in your Terraform configurations
  </Card>

  <Card
    title="Pulumi"
    href="/template-catalog/install-manage/pulumi"
    icon={<svg xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 24 24" id="Pulumi-Icon--Streamline-Svg-Logos" height="24" width="24">
    <desc>
        Pulumi Icon Streamline Icon: https://streamlinehq.com
    </desc>
    <path fill="#f26e7e" d="M4.683025 13.3318c0.869125 -0.5018 0.870575 -2.1264 0.003225 -3.62865s-2.27504 -2.313275 -3.1441725 -1.811475C0.672945 8.3935 0.6715 10.0181 1.53885 11.52035c0.86735 1.502275 2.27505 2.313275 3.144175 1.81145Zm0.0052 3.2167c0.86735 1.502275 0.865925 3.126875 -0.003225 3.628675 -0.86915 0.5018 -2.2768275 -0.309225 -3.144175 -1.81145 -0.8673525 -1.50225 -0.8659075 -3.126875 0.003225 -3.628675 0.8691325 -0.5018 2.276825 0.309225 3.144175 1.81145Zm5.922875 3.4243c0.86735 1.50225 0.8659 3.126775 -0.003225 3.62875 -0.869125 0.501775 -2.27685 -0.309325 -3.1442 -1.81155 -0.867325 -1.50225 -0.865875 -3.12685 0.00325 -3.628675 0.869125 -0.5018 2.276825 0.309225 3.144175 1.811475Zm-0.001925 -6.845275c0.86735 1.50225 0.8659 3.12685 -0.003225 3.628675 -0.869125 0.5018 -2.276825 -0.309225 -3.144175 -1.811475 -0.86735 -1.50225 -0.8659 -3.12685 0.003225 -3.62865 0.869125 -0.501825 2.276825 0.3092 3.144175 1.81145Z" stroke-width="0.25"></path>
    <path fill="#8a3391" d="M22.45775 11.524125c0.86725 -1.502225 0.865925 -3.12685 -0.003225 -3.62865 -0.869125 -0.501825 -2.276825 0.3092 -3.144175 1.811475 -0.86735 1.50225 -0.8659 3.126825 0.003225 3.62865 0.869125 0.501825 2.276825 -0.3092 3.144175 -1.811475Zm0.000175 3.2151c0.869075 0.5018 0.870625 2.1264 0.003225 3.62865 -0.86735 1.50225 -2.27505 2.313275 -3.144175 1.81145 -0.869125 -0.5018 -0.870575 -2.126425 -0.003225 -3.62865 0.86735 -1.50225 2.27505 -2.313275 3.144175 -1.81145ZM16.536225 18.157875c0.86915 0.501825 0.8706 2.126425 0.00325 3.628675 -0.86735 1.502125 -2.275075 2.313225 -3.1442 1.81145 -0.869125 -0.50175 -0.870575 -2.126425 -0.003225 -3.62865 0.867375 -1.502275 2.27505 -2.3133 3.144175 -1.811475Zm-0.003325 -6.843775c0.869125 0.5018 0.870575 2.126425 0.003225 3.628675s-2.27505 2.313275 -3.1442 1.811475c-0.869125 -0.501825 -0.870575 -2.126425 -0.003225 -3.628675 0.86735 -1.502275 2.27505 -2.313275 3.1442 -1.811475Z" stroke-width="0.25"></path>
    <path fill="#f7bf2a" d="M15.138225 2.06721c0 1.003615 -1.40625 1.817215 -3.14095 1.817215 -1.7347 0 -3.14095 -0.8136 -3.14095 -1.817215C8.856325 1.06359 10.262575 0.25 11.997275 0.25c1.7347 0 3.14095 0.81359 3.14095 1.81721ZM9.2166 5.482375c0 1.003625 -1.40625 1.8172 -3.14095 1.8172 -1.7347 0 -3.14095 -0.813575 -3.14095 -1.8172s1.40625 -1.817225 3.14095 -1.817225c1.7347 0 3.14095 0.8136 3.14095 1.817225Zm8.71005 1.8172c1.7347 0 3.14095 -0.813575 3.14095 -1.8172s-1.40625 -1.817225 -3.14095 -1.817225c-1.7347 0 -3.14095 0.8136 -3.14095 1.817225s1.40625 1.8172 3.14095 1.8172Zm-2.788425 1.605625c0 1.003625 -1.40625 1.8172 -3.14095 1.8172 -1.7347 0 -3.14095 -0.813575 -3.14095 -1.8172 0 -1.0036 1.40625 -1.8172 3.14095 -1.8172 1.7347 0 3.14095 0.8136 3.14095 1.8172Z" stroke-width="0.25"></path>
    </svg>}
  >
    Declare templates in your Pulumi programs
  </Card>
</CardGroup>

## Configuration

The default `values.yaml` for this template:

```yaml theme={null}
# ─── GVC and locations ────────────────────────────────────────────────────────
# Lives under `global` so a parent chart (postgres-multi-location) sets it once
# and Helm propagates it here. See README before changing.

global:
  gvc:
    name: etcd-multi-location-gvc # the GVC this chart creates
    # One etcd member per location. Minimum 2; 3 survives losing one location,
    # 5 survives losing two. See the quorum table in the README.
    locations:
      - name: aws-us-east-1
      - name: aws-eu-central-1
      - name: aws-us-west-2

# ─── etcd ─────────────────────────────────────────────────────────────────────
image: controlplanecorporation/etcd:0.1

resources:
  cpu: 500m
  memory: 512Mi

# Raft timers, tuned for cross-region round trips (measured 63-140 ms).
# Raise both if your locations are more than ~250 ms apart.
tuning:
  heartbeatIntervalMs: 250 # ~0.5-1.5x the worst round trip between locations
  electionTimeoutMs: 5000 # must be >= 10x heartbeatIntervalMs; maximum 50000

volumeset:
  capacity: 10 # initial capacity in GiB (minimum is 10)

internalAccess:
  type: same-gvc # options: same-gvc, same-org, workload-list
  workloads: [] # only used when type is workload-list
  #- //gvc/GVC_NAME/workload/WORKLOAD_NAME

# ─── Disaster recovery ────────────────────────────────────────────────────────
recovery:
  # EMERGENCY ONLY. Set to the surviving location's name to restart that member
  # as a new single-member cluster after permanently losing quorum. Read the
  # "Recovering from a lost location" section of the README first.
  forceNewClusterInLocation: ""
```

### Locations and Quorum

* `global.gvc.name` — Name of the GVC this template creates. It must **not** name a GVC that already exists.
* `global.gvc.locations` — The member map: one etcd member is deployed in each listed location. At least 2 locations are required.

<Warning>
  Helm owns the GVC this template creates. If `global.gvc.name` matches a GVC that already exists, Helm adopts it and a later uninstall deletes that GVC along with everything inside it. Always pick a name no other release uses.
</Warning>

etcd commits a write only when a **majority** of members accept it. Because there is exactly one member per location, the location is the failure domain:

| Locations | Majority needed | Location losses survived | What that means                                                                          |
| --------- | --------------- | ------------------------ | ---------------------------------------------------------------------------------------- |
| 2         | 2               | **0**                    | Losing either one stops writes. The survivor holds current data, but recovery is manual. |
| 3         | 2               | **1**                    | Automatic failover. The recommended shape.                                               |
| 4         | 3               | **1**                    | No better than 3, and costs more.                                                        |
| 5         | 3               | **2**                    | Survives losing two locations.                                                           |

With N locations you survive `floor((N-1)/2)` losses, so an even count never buys anything over the odd count below it. Two locations is permitted as a deliberate warm-standby topology, but it survives nothing automatically.

The `global.gvc` block lives under `global` so that a parent chart — `postgres-multi-location` consumes this template as a subchart — sets the GVC and location list once and Helm propagates it. Each location runs exactly one member; any other replica count fails at render time.

<Note>
  Changing `global.gvc.locations` reprovisions the cluster: every member restarts with a new cluster list. This is not etcd's graceful `member add` / `member remove` path, so plan it as a maintenance window.
</Note>

### Image and Resources

* `image` — The etcd image to run (etcd 3.6.5).
* `resources.cpu` / `resources.memory` — CPU and memory allocated to each member (default: `500m`, `512Mi`).

### Raft Tuning

* `tuning.heartbeatIntervalMs` — Leader heartbeat interval, roughly 0.5–1.5× the worst round trip between your locations.
* `tuning.electionTimeoutMs` — How long a follower waits before campaigning. Must be at least 10× the heartbeat interval, and at most `50000`. Both bounds are enforced at render time.

The defaults detect a dead leader in about 5 seconds across an AWS us-east ↔ eu-central ↔ us-west triangle. Locations further apart (US ↔ Asia-Pacific is 350–400 ms) need both values raised in proportion.

### Storage

* `volumeset.capacity` — Persistent volume size in GiB per member for the raft write-ahead log and snapshots (minimum 10).

Auto-compaction is enabled by design (periodic, every hour) and is not configurable. The first compaction happens an hour after a member starts, and a restart resets that clock. Without it, a continuously written cluster grows revisions until it reaches etcd's 2 GiB backend quota and goes read-only. Compaction reclaims pages for reuse rather than shrinking the file, so the reported `dbSize` plateaus instead of dropping — only `etcdctl defrag` returns space to the filesystem.

### Internal Access

The `internalAccess` section controls which workloads can reach the client API on `2379`. Cross-location traffic inside one GVC is same-GVC traffic, so the default covers a stretched cluster with no extra rule.

| Type            | Description                                                                        |
| --------------- | ---------------------------------------------------------------------------------- |
| `same-gvc`      | Allow access from all workloads in the same GVC (recommended)                      |
| `same-org`      | Allow access from all workloads in the same organization                           |
| `workload-list` | Allow access only from the specific workloads listed in `internalAccess.workloads` |

<Warning>
  There is deliberately no public access: etcd in this template runs without TLS and without authentication. Anything permitted by `internalAccess` has full read/write access to the entire keyspace. Use `workload-list` if the GVC contains workloads that should not have it.
</Warning>

Firewall changes take up to roughly two and a half minutes to take effect. Re-test after waiting rather than concluding the setting was ignored.

### Connecting to etcd

| Target                                     | Address                                                         |
| ------------------------------------------ | --------------------------------------------------------------- |
| Client API, load-balanced across locations | `RELEASE_NAME-etcd.GVC_NAME.cpln.local:2379`                    |
| A specific location's member               | `replica-0.RELEASE_NAME-etcd.LOCATION.GVC_NAME.cpln.local:2379` |
| Raft peer traffic (members only)           | Port `2380`                                                     |

Point clients at one endpoint per location so they can fail over. From inside the cluster:

```bash theme={null}
cpln workload exec RELEASE_NAME-etcd --gvc GVC_NAME --container etcd -- etcdctl member list -w table
cpln workload exec RELEASE_NAME-etcd --gvc GVC_NAME --container etcd -- etcdctl endpoint status --cluster -w table
cpln workload exec RELEASE_NAME-etcd --gvc GVC_NAME --container etcd -- etcdctl put /demo/key hello
cpln workload exec RELEASE_NAME-etcd --gvc GVC_NAME --container etcd -- etcdctl --command-timeout=5s get /demo/key
```

Every member is named `RELEASE_NAME-etcd-LOCATION`, so `member list` maps one-to-one onto your location list.

<Note>
  Allow about two minutes of convergence after a cold install before concluding that a member is unreachable — cross-region service discovery can lag a workload reporting ready by well over a minute.
</Note>

### Ports

| Port   | Protocol | Description                       |
| ------ | -------- | --------------------------------- |
| `2379` | TCP      | etcd client API (internal only)   |
| `2380` | TCP      | Raft peer traffic between members |

## Failover Behavior

Measured on a three-location cluster (`aws-us-east-1`, `aws-eu-central-1`, `aws-us-west-2`) with the default 5 s election timeout:

| Event                | Result                                                                                            |
| -------------------- | ------------------------------------------------------------------------------------------------- |
| Idle stability       | 10.8 hours with zero spontaneous elections                                                        |
| Leader crash         | New leader elected in **5.59 s**, with 0 failed writes on a client pinned to a surviving follower |
| Planned replica stop | Leadership handed over in **502 ms**, 0 failed writes across 570 samples                          |

Write latency is bounded by one cross-region round trip, because the leader needs a follower acknowledgement before it can commit. Measured with the leader in `aws-eu-central-1`:

| Client location                                 | Write latency (p50) |
| ----------------------------------------------- | ------------------- |
| `aws-eu-central-1` (co-located with the leader) | 95.7 ms             |
| `aws-us-east-1`                                 | 185.7 ms            |
| `aws-us-west-2` (furthest from the leader)      | 236.4 ms            |

A client co-located with the leader does not get local-write latency — one cross-region round trip is the floor for any stretched quorum.

## Upgrades

<Warning>
  A `helm upgrade` takes the whole cluster down for about **66 seconds** (measured on three locations). Members in every location restart together, quorum is lost, and writes time out until it returns. The cluster recovers on its own, but every configuration change — including one that only changes a firewall rule — costs that window, so plan upgrades as a short planned outage.
</Warning>

Nothing serializes the restart: the field that would limit it (`rolloutOptions.maxUnavailableReplicas`) is not retained by the platform, so the chart deliberately does not set it. Running more members per location has not been tested and should not be assumed to help.

## Behavior Under Quorum Loss

When a majority of members is unreachable, the cluster stops committing writes. What that looks like from a client is easy to misread:

* **Writes hang rather than failing fast.** They block until they time out, so always give clients a short `--command-timeout` (or the client-library equivalent) — otherwise connections pile up against a cluster that cannot commit.
* **Serializable reads keep succeeding against stale data.** A read issued with `--consistency=s` is served from the local member's own store and never notices the loss of quorum.
* **`IS LEADER: true` is not proof of leadership.** An isolated survivor keeps reporting itself as leader for about 6 seconds while unable to commit anything. Health-check with a linearizable read (`etcdctl get KEY`, without `--consistency=s`), never with `endpoint status`.

<Warning>
  Never suspend a location for this workload. Suspending and resuming a location permanently withdraws that location's endpoints from the other locations' service discovery while every status surface still reports healthy. Add or remove locations by editing `global.gvc.locations` instead.
</Warning>

## Recovering from a Lost Location

`recovery.forceNewClusterInLocation` is for the case where quorum is **permanently** gone — with two locations, that is the loss of either one. It is not needed for a location that is coming back: a member that returns with its volume intact rejoins on its own in well under a minute.

<Steps>
  <Step title="Confirm the loss is permanent">
    `etcdctl member remove` cannot help here, because removing a member itself requires quorum.
  </Step>

  <Step title="Force a new single-member cluster">
    Set `recovery.forceNewClusterInLocation` to the **surviving** location's name and upgrade the release. That member restarts as a single-member cluster rebuilt from its own write-ahead log and serves writes again immediately.
  </Step>

  <Step title="Clear the setting">
    Set `recovery.forceNewClusterInLocation` back to `""` and upgrade again. Leaving it set means the flag fires on every future restart of that member.
  </Step>

  <Step title="Reset the lost location's volume before it rejoins">
    The evicted members still hold the old cluster ID and refuse to start until their data directory is cleared. Uninstall and reinstall, or delete that volume, so the member bootstraps fresh.
  </Step>
</Steps>

<Warning>
  Never set `recovery.forceNewClusterInLocation` to more than one location, and never leave it set. Two members both forcing a new cluster produce two divergent single-member clusters that cannot be merged.
</Warning>

## External References

<CardGroup cols={2}>
  <Card title="etcd Documentation" icon="book" href="https://etcd.io/docs/v3.6/">
    Official etcd documentation
  </Card>

  <Card title="Tuning for Latency" icon="gauge" href="https://etcd.io/docs/v3.6/tuning/">
    Heartbeat and election timeout guidance for cross-region clusters
  </Card>

  <Card title="Disaster Recovery" icon="life-ring" href="https://etcd.io/docs/v3.6/op-guide/recovery/">
    Recovering an etcd cluster that has lost quorum
  </Card>

  <Card title="etcd Multi-Location Template" icon="github" href="https://github.com/controlplane-com/templates/tree/main/etcd-multi-location">
    View the source files, default values, and chart definition
  </Card>
</CardGroup>