> ## Documentation Index
> Fetch the complete documentation index at: https://docs.controlplane.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Grafana Multi-Location

> Grafana OSS in every location of an existing multi-location GVC behind one georouted endpoint, with all state in a bundled stretched PostgreSQL cluster and alerting run by one pinned evaluator or Redis-coordinated across every instance. Three prerequisite secrets must exist before install.

## Overview

Grafana Multi-Location deploys **one logical Grafana** whose UI/API instances run in every location you configure, behind a single georouted `*.cpln.app` endpoint. Every dashboard, user, org, session, alert rule and saved datasource lives in a bundled [`postgres-multi-location`](/template-catalog/templates/postgres-multi-location) cluster stretched across the same locations, so Grafana itself holds no state — there is no volume, no session affinity and nothing to hand over when an instance is replaced.

Alerting comes in two shapes, chosen with `alerting.highAvailability.enabled`: a **single dedicated evaluator** pinned to one location (the default), or **every instance evaluating** with a stretched Redis tier making sure exactly one of them sends each notification. See [Alert Evaluation](#alert-evaluation) before choosing.

<Note>
  This template deploys into a GVC **you already have** — it does not create, provision or manage one. That GVC must have at least 2 locations (3 with alerting HA enabled), and every location you list in `global.locations` must already be one of them. GVC locations you do not list are left alone: nothing from this release runs there. For a Grafana inside a single location, use the [Grafana](/template-catalog/templates/grafana) template instead.
</Note>

### What Gets Created

| Resource | Name | Purpose | Optional |
| - | - | - | - |
| Workload | `RELEASE_NAME-grafana` | Grafana UI/API tier (standard, `replicas` instances per location, HTTP port `3000`, public by default) | — |
| Workload | `RELEASE_NAME-grafana-alerting` | Dedicated alert evaluator (standard, exactly 1 replica, in `alerting.location` only, never public) | Default; not created when `alerting.highAvailability.enabled: true` or `alerting.enabled: false` |
| Workload | `RELEASE_NAME-postgres` | Patroni PostgreSQL app database (stateful, `global.locations[].replicas` members per location) | — |
| Workload | `RELEASE_NAME-postgres-proxy` | HAProxy leader-routing tier (standard, per location, always points at the current primary) | — |
| Workload | `RELEASE_NAME-etcd` | Consensus store for the database (stateful, exactly 1 member per location) | — |
| Workload | `RELEASE_NAME-postgres-backup` | Scheduled logical database backup (cron, one location) | `postgresML.backup.enabled: true` with `mode: logical` |
| Workload | `RELEASE_NAME-redis`, `RELEASE_NAME-sentinel` | Alerting coordination tier (stateful, one Sentinel and `redisML.redis.replicasPerLocation` Redis members per location) | `alerting.highAvailability.enabled: true` |
| Volume set | `RELEASE_NAME-postgres-vs`, `RELEASE_NAME-etcd-vs` | Database and consensus-store data (10 GiB per member by default) | — |
| Volume set | `RELEASE_NAME-redis-vs`, `RELEASE_NAME-sentinel-vs` | Coordination-tier data | `alerting.highAvailability.enabled: true` |
| Secret | `RELEASE_NAME-grafana-datasources` | Rendered datasource provisioning file, mounted by every Grafana workload | `datasources.definitions` non-empty |
| Secret | `RELEASE_NAME-postgres-startup`, `RELEASE_NAME-postgres-proxy-startup`, `RELEASE_NAME-etcd-startup` | Startup scripts of the database tier | — |
| Secret | `RELEASE_NAME-redis-config`, `RELEASE_NAME-sentinel-config` | Coordination-tier configuration | `alerting.highAvailability.enabled: true` |
| Identity | `RELEASE_NAME-grafana-identity` | Shared by both Grafana workloads | — |
| Policy | `RELEASE_NAME-grafana-policy` | `reveal` on exactly the secrets the Grafana workloads mount | — |
| Policy | `RELEASE_NAME-grafana-gvc-policy` | `view` on the one GVC this release installs into, so each instance can check the GVC's location list at boot | — |
| Identity / Policy | `RELEASE_NAME-postgres-identity`, `RELEASE_NAME-postgres-policy`, `RELEASE_NAME-postgres-gvc-policy`, `RELEASE_NAME-etcd-identity`, `RELEASE_NAME-etcd-policy`, `RELEASE_NAME-etcd-gvc-policy` | The database tier's own identities and policies | — |
| Identity / Policy | `RELEASE_NAME-redis-identity`, `RELEASE_NAME-redis-policy`, `RELEASE_NAME-redis-gvc-policy`, `RELEASE_NAME-sentinel-identity`, `RELEASE_NAME-sentinel-policy` | The coordination tier's own identities and policies | `alerting.highAvailability.enabled: true` |

The admin password, the encryption key and the database credentials are **prerequisite secrets you create yourself**; the chart references them by name and never creates, modifies or deletes them.

## Prerequisites

<Steps>
  <Step title="Have a GVC with at least 2 locations">
    This template creates no GVC. Read the locations of the GVC you will install into and make every entry of `global.locations` one of them:

    ```bash theme={null}
    cpln gvc get GVC_NAME -o yaml
    ```

    Compare `spec.staticPlacement.locationLinks` with your values. The database commits a write only when a **majority** of its members agree, and it runs one member per location, so the location count decides what survives:

    | Locations | Location losses survived | What a user sees when one location is lost |
    | - | - | - |
    | **2** | **0** | Dashboards may still render, but logins, saves and alert-state writes fail until the survivor is promoted by hand. |
    | **3** | **1** | Automatic database failover; the endpoint keeps serving from the two surviving locations. |
    | **5** | **2** | Survives losing two locations. |

    With N locations you survive `floor((N-1)/2)` losses, so an even count buys nothing over the odd count below it. `alerting.highAvailability.enabled: true` requires at least **3** locations, and the chart refuses to render below that.
  </Step>

  <Step title="Create the first-boot admin password">
    An [opaque secret](/guides/create-secret/opaque) with encoding `plain`. Grafana reads it only when the admin account is first created:

    ```bash theme={null}
    printf '%s' "$(openssl rand -hex 24)" | cpln secret create-opaque --name ADMIN_PASSWORD_SECRET_NAME --encoding plain -f -
    ```

    Set `admin.passwordSecretName` to this name.
  </Step>

  <Step title="Create the encryption key">
    An [opaque secret](/guides/create-secret/opaque) with encoding `plain`. Every instance in every location reads it on every boot to decrypt datasource credentials stored in the shared database — back it up outside Control Plane and never rotate it:

    ```bash theme={null}
    printf '%s' "$(openssl rand -hex 32)" | cpln secret create-opaque --name SECRET_KEY_SECRET_NAME --encoding plain -f -
    ```

    Set `admin.secretKeySecretName` to this name.
  </Step>

  <Step title="Create the app database credentials">
    A [dictionary secret](/guides/create-secret/dictionary) holding exactly `username`, `password` and `database`. Use plain identifiers for `username` and `database` — they are used unquoted when the database is created:

    ```bash theme={null}
    cpln secret create-dictionary --name DB_CREDENTIALS_SECRET_NAME \
      --entry username=grafana \
      --entry password="$(openssl rand -hex 24)" \
      --entry database=grafana
    ```

    Set `postgresML.postgres.credentialsSecretName` to this name.
  </Step>
</Steps>

<Warning>
  **A missing prerequisite secret wedges the install rather than failing it.** `cpln helm install` reports success while the affected workloads sit at zero replicas with no containers and therefore no logs. The only place the missing secret is named is `status.versions[].message`:

  ```bash theme={null}
  cpln workload get-deployments RELEASE_NAME-grafana --gvc GVC_NAME -o yaml
  ```

  Create the missing secret and the deployment recovers on its own after several minutes, or run `cpln workload force-redeployment RELEASE_NAME-grafana --gvc GVC_NAME` to skip the wait.
</Warning>

Optional features each need something created **before** you install:

| Feature | What you must create first |
| - | - |
| Credentialed provisioned datasources | One [dictionary secret](/guides/create-secret/dictionary) per entry in `datasources.credentialSecrets` — see [Datasources as Code](#datasources-as-code) |
| Authenticated SMTP | An [opaque secret](/guides/create-secret/opaque) with encoding `plain` holding the SMTP password, named in `smtp.passwordSecretName` |
| Authenticated alerting-HA Redis tier | One [opaque secret](/guides/create-secret/opaque) with encoding `plain` per password — see [Authenticating the Redis Tier](#authenticating-the-redis-tier). Alerting HA itself needs **no** extra secrets |
| Database backups | A bucket and access setup on AWS S3, Google Cloud Storage or an S3-compatible server — see [Backing Up](#backing-up) |

## Installation

Once the three secrets exist, install the template into the GVC with the required secret names:

```bash theme={null}
cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/grafana-multi-location \
  --version 2.0.0 --gvc GVC_NAME \
  --set admin.passwordSecretName=ADMIN_PASSWORD_SECRET_NAME \
  --set admin.secretKeySecretName=SECRET_KEY_SECRET_NAME \
  --set postgresML.postgres.credentialsSecretName=DB_CREDENTIALS_SECRET_NAME
```

The default `global.locations` list is `aws-us-east-1`, `aws-eu-central-1` and `aws-us-west-2`. If your GVC's locations differ, supply your own list in a values file (`--values`) alongside the command above, and keep `alerting.location` and `postgresML.primaryLocation` pointing at one of the locations you list.

<CardGroup cols={2}>
  <Card title="UI" href="/template-catalog/install-manage/ui" icon="laptop">
    Browse, install, and manage templates visually
  </Card>

  <Card title="CLI" href="/template-catalog/install-manage/cli" icon="terminal">
    Manage templates from your terminal
  </Card>

  <Card title="Terraform" href="/template-catalog/install-manage/terraform" icon={<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 128 128"><g fill-rule="evenodd"><path d="M77.941 44.5v36.836L46.324 62.918V26.082zm0 0" fill="#5c4ee5"/><path d="M81.41 81.336l31.633-18.418V26.082L81.41 44.5zm0 0" fill="#4040b2"/><path d="M11.242 42.36L42.86 60.776V23.941L11.242 5.523zm0 0M77.941 85.375L46.324 66.957v36.82l31.617 18.418zm0 0" fill="#5c4ee5"/></g></svg>}>
    Declare templates in your Terraform configurations
  </Card>

  <Card
    title="Pulumi"
    href="/template-catalog/install-manage/pulumi"
    icon={<svg xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 24 24" id="Pulumi-Icon--Streamline-Svg-Logos" height="24" width="24">
    <desc>
        Pulumi Icon Streamline Icon: https://streamlinehq.com
    </desc>
    <path fill="#f26e7e" d="M4.683025 13.3318c0.869125 -0.5018 0.870575 -2.1264 0.003225 -3.62865s-2.27504 -2.313275 -3.1441725 -1.811475C0.672945 8.3935 0.6715 10.0181 1.53885 11.52035c0.86735 1.502275 2.27505 2.313275 3.144175 1.81145Zm0.0052 3.2167c0.86735 1.502275 0.865925 3.126875 -0.003225 3.628675 -0.86915 0.5018 -2.2768275 -0.309225 -3.144175 -1.81145 -0.8673525 -1.50225 -0.8659075 -3.126875 0.003225 -3.628675 0.8691325 -0.5018 2.276825 0.309225 3.144175 1.81145Zm5.922875 3.4243c0.86735 1.50225 0.8659 3.126775 -0.003225 3.62875 -0.869125 0.501775 -2.27685 -0.309325 -3.1442 -1.81155 -0.867325 -1.50225 -0.865875 -3.12685 0.00325 -3.628675 0.869125 -0.5018 2.276825 0.309225 3.144175 1.811475Zm-0.001925 -6.845275c0.86735 1.50225 0.8659 3.12685 -0.003225 3.628675 -0.869125 0.5018 -2.276825 -0.309225 -3.144175 -1.811475 -0.86735 -1.50225 -0.8659 -3.12685 0.003225 -3.62865 0.869125 -0.501825 2.276825 0.3092 3.144175 1.81145Z" stroke-width="0.25"></path>
    <path fill="#8a3391" d="M22.45775 11.524125c0.86725 -1.502225 0.865925 -3.12685 -0.003225 -3.62865 -0.869125 -0.501825 -2.276825 0.3092 -3.144175 1.811475 -0.86735 1.50225 -0.8659 3.126825 0.003225 3.62865 0.869125 0.501825 2.276825 -0.3092 3.144175 -1.811475Zm0.000175 3.2151c0.869075 0.5018 0.870625 2.1264 0.003225 3.62865 -0.86735 1.50225 -2.27505 2.313275 -3.144175 1.81145 -0.869125 -0.5018 -0.870575 -2.126425 -0.003225 -3.62865 0.86735 -1.50225 2.27505 -2.313275 3.144175 -1.81145ZM16.536225 18.157875c0.86915 0.501825 0.8706 2.126425 0.00325 3.628675 -0.86735 1.502125 -2.275075 2.313225 -3.1442 1.81145 -0.869125 -0.50175 -0.870575 -2.126425 -0.003225 -3.62865 0.867375 -1.502275 2.27505 -2.3133 3.144175 -1.811475Zm-0.003325 -6.843775c0.869125 0.5018 0.870575 2.126425 0.003225 3.628675s-2.27505 2.313275 -3.1442 1.811475c-0.869125 -0.501825 -0.870575 -2.126425 -0.003225 -3.628675 0.86735 -1.502275 2.27505 -2.313275 3.1442 -1.811475Z" stroke-width="0.25"></path>
    <path fill="#f7bf2a" d="M15.138225 2.06721c0 1.003615 -1.40625 1.817215 -3.14095 1.817215 -1.7347 0 -3.14095 -0.8136 -3.14095 -1.817215C8.856325 1.06359 10.262575 0.25 11.997275 0.25c1.7347 0 3.14095 0.81359 3.14095 1.81721ZM9.2166 5.482375c0 1.003625 -1.40625 1.8172 -3.14095 1.8172 -1.7347 0 -3.14095 -0.813575 -3.14095 -1.8172s1.40625 -1.817225 3.14095 -1.817225c1.7347 0 3.14095 0.8136 3.14095 1.817225Zm8.71005 1.8172c1.7347 0 3.14095 -0.813575 3.14095 -1.8172s-1.40625 -1.817225 -3.14095 -1.817225c-1.7347 0 -3.14095 0.8136 -3.14095 1.817225s1.40625 1.8172 3.14095 1.8172Zm-2.788425 1.605625c0 1.003625 -1.40625 1.8172 -3.14095 1.8172 -1.7347 0 -3.14095 -0.813575 -3.14095 -1.8172 0 -1.0036 1.40625 -1.8172 3.14095 -1.8172 1.7347 0 3.14095 0.8136 3.14095 1.8172Z" stroke-width="0.25"></path>
    </svg>}
  >
    Declare templates in your Pulumi programs
  </Card>
</CardGroup>

## Configuration

### Locations

```yaml theme={null}
# Every location listed here MUST already exist in the GVC you install into.
# Extra locations in the GVC are fine — nothing here runs in them.
# Minimum 2 locations, and minimum 3 with alerting HA enabled.
# `replicas` here sets the POSTGRES members in that location, and nothing else:
#   etcd     — always exactly 1 per location, not configurable.
#   Grafana  — the top-level `replicas` below, applied to every location.
#   Redis    — redisML.redis.replicasPerLocation (alerting HA only).
global:
  locations:
    - name: aws-us-east-1
      replicas: 1
    - name: aws-eu-central-1
      replicas: 1
    - name: aws-us-west-2
      replicas: 1
```

The list lives under `global` so the bundled database, its consensus store and the optional Redis tier all receive the same list — there is never a second list to maintain. Removing a location from the list is the supported way to shrink the deployment; `replicas` must stay at least 1.

Both Grafana tiers read the GVC at boot and log a `[grafana] WARNING:` line naming any location in `global.locations` that the GVC does not have. See [Troubleshooting](#troubleshooting) for what that means for alerting.

### Grafana UI Tier

```yaml theme={null}
image: grafana/grafana:13.1.3

replicas: 1 # Grafana UI instances PER LOCATION

resources:
  maxCpu: 1000m
  maxMemory: 1Gi
  minCpu: 500m
  minMemory: 512Mi

database:
  maxOpenConn: 10 # per instance; see the connection budget below
```

<Warning>
  `(replicas × locations + 1) × database.maxOpenConn` must stay at **80 or less**; the `+ 1` is the dedicated alert evaluator and is dropped when there is none (alerting HA on, or `alerting.enabled: false`). The chart enforces this at render time and refuses to install with the arithmetic spelled out, so raising `replicas` past that point means lowering `database.maxOpenConn`.
</Warning>

`image` must provide `/bin/bash` — the chart wraps Grafana's own entrypoint in a boot script; the default official image does. Signup, anonymous access and upstream analytics are disabled by the template and are not configurable. With alerting HA on, rule evaluation runs on this tier, so a large rule set may need `resources.maxCpu` raised.

### Alerting

```yaml theme={null}
alerting:
  enabled: true # false = rules can be created and viewed, nothing ever evaluates them
  highAvailability:
    enabled: false # true = every UI instance evaluates, Redis coordinates the send; needs 3+ locations
  # Dedicated-evaluator mode ONLY — where the one evaluator runs. Must be one of
  # global.locations. IGNORED when highAvailability.enabled is true.
  location: aws-us-east-1
  # Dedicated-evaluator mode ONLY — in HA mode the UI tier's `resources` apply.
  resources:
    maxCpu: 1000m
    maxMemory: 1Gi
    minCpu: 500m
    minMemory: 512Mi
```

Read [Alert Evaluation](#alert-evaluation) before enabling `highAvailability` — it multiplies your data-source query load by the number of Grafana instances. `alerting.enabled: false` combined with `highAvailability.enabled: true` fails at render time.

### Admin Credentials and Encryption Key

```yaml theme={null}
admin:
  user: admin # initial admin login name (not sensitive)
  applyPassword: true # set false after your first login to stop referencing the password secret
  passwordSecretName: my-grafana-admin-password # opaque secret holding the first-boot admin password
  secretKeySecretName: my-grafana-secret-key # opaque secret holding the at-rest encryption key
```

Both are [opaque secrets](/guides/create-secret/opaque) you create before installing (see [Prerequisites](#prerequisites)); neither value ever passes through Helm. Their lifecycles are opposites:

| | `admin.passwordSecretName` | `admin.secretKeySecretName` |
| - | - | - |
| What it holds | The admin login password | The key encrypting stored datasource credentials |
| When Grafana reads it | Only when the admin account is **first created** | On **every boot**, on every instance |
| Can you stop referencing it? | Yes — set `admin.applyPassword: false` once you have logged in | **No** |
| Can you rotate or delete it? | Change the password in the UI, then set `applyPassword: false` and delete the secret | **Never** — every stored datasource credential becomes unreadable in every location |

### Datasources as Code

```yaml theme={null}
datasources:
  definitions: []
  # - name: Prometheus
  #   type: prometheus
  #   access: proxy
  #   # Substitute BOTH parts: the workload name AND the GVC.
  #   url: http://YOUR_WORKLOAD.YOUR_GVC.cpln.local:9095
  #   isDefault: true
  credentialSecrets: []
  # - name: my-grafana-ds-credentials
  #   keys: [PG_PASSWORD]
```

`definitions` are standard [Grafana datasource provisioning](https://grafana.com/docs/grafana/latest/administration/provisioning/#data-sources) entries, rendered verbatim into a file that every Grafana instance applies at boot. The file is plaintext, so credentials never go into it: create a [dictionary secret](/guides/create-secret/dictionary) **before installing**, list it under `credentialSecrets`, and write `$KEY` in the definition. Each listed key becomes an environment variable the chart grants the identity `reveal` on, and Grafana interpolates it when it loads the file:

```bash theme={null}
cpln secret create-dictionary --name DATASOURCE_CREDENTIALS_SECRET_NAME \
  --entry PG_PASSWORD='YOUR-DATASOURCE-PASSWORD'
```

```yaml theme={null}
datasources:
  definitions:
    - name: AppDB
      type: postgres
      url: my-db-host:5432
      user: grafana_reader
      jsonData: { database: appdb, sslmode: disable }
      secureJsonData:
        password: $PG_PASSWORD
  credentialSecrets:
    - name: DATASOURCE_CREDENTIALS_SECRET_NAME
      keys: [PG_PASSWORD]
```

Provisioned datasources are read-only in the UI — change the values and upgrade to change one.

### SMTP

```yaml theme={null}
smtp:
  enabled: false
  host: smtp.example.com:587 # host:port
  user: "" # empty = unauthenticated SMTP; if set, the relay MUST offer STARTTLS or TLS
  passwordSecretName: "" # opaque secret (encoding: plain) with the SMTP password; create BEFORE install
  fromAddress: grafana@example.com
  fromName: Grafana
```

Whichever instance evaluates is what sends: the dedicated evaluator by default, or the coordinating UI instance with alerting HA on. `smtp.passwordSecretName` is required whenever `smtp.user` is set:

```bash theme={null}
printf '%s' 'YOUR-SMTP-PASSWORD' | cpln secret create-opaque --name SMTP_PASSWORD_SECRET_NAME --encoding plain -f -
```

<Warning>
  **Authenticated SMTP requires a relay that offers STARTTLS or TLS.** Grafana refuses to send credentials over an unencrypted connection, fails with `failed to send email: unencrypted connection`, and every notification is lost — the only signal is a log line in the sending workload. Hosted relays (SES, SendGrid, Mailgun, Microsoft 365, Gmail) are fine; a plain in-GVC relay is not. Leave `smtp.user` empty for such relays.
</Warning>

### Access

```yaml theme={null}
publicAccess:
  enabled: true # UI on the canonical *.cpln.app HTTPS endpoint

internalAccess:
  type: same-gvc # options: none, same-gvc, same-org, workload-list
  # Only used when type is workload-list. This release's OWN workloads are added
  # automatically — list your clients here, not this release's workloads.
  workloads: []
  # workloads:
  #   - //gvc/GVC_NAME/workload/WORKLOAD_NAME
```

`publicAccess` applies to the UI tier only; with it on, links in alert notifications point at the public endpoint. The dedicated alert evaluator is never reachable from the internet and honours `internalAccess` only. With alerting HA on there is no evaluator, and the Redis tier is reachable from inside the GVC only.

<Warning>
  **The bundled database and Redis tiers keep their own firewall lists, and the Grafana workloads are not added to them automatically.** Setting `postgresML.internalAccess.type` or `redisML.firewall.internalAllowType` to `workload-list` without listing the Grafana workloads would cut Grafana off from its database or its alerting coordinator, so the chart refuses to render in that state and prints the exact links to paste into the subchart's `workloads` list.
</Warning>

Allow several minutes after changing an access knob before concluding it did not apply.

### App Database

```yaml theme={null}
postgresML:
  etcd:
    tuning:
      autoCompactionMode: periodic # periodic (retention is a duration) or revision (a revision count)
      autoCompactionRetention: 1h # periodic needs an explicit unit (1h, 30m, 24h)
      quotaBackendBytes: 0 # backend size limit in bytes; 0 = etcd's own default of 2 GiB

  postgres:
    credentialsSecretName: my-grafana-db-credentials # REQUIRED prerequisite dictionary secret

  internalAccess:
    type: same-gvc # options: same-gvc, same-org, workload-list
    workloads: []

  # Preferred location for the database primary. Keep it aligned with
  # alerting.location so the hot path has no cross-region hop.
  primaryLocation: aws-us-east-1

  resources:
    minCpu: 500m
    minMemory: 1Gi
    maxCpu: 1
    maxMemory: 2Gi

  volumeset:
    capacity: 10 # initial capacity in GiB per member (minimum is 10)

  proxy:
    minReplicas: 2 # HAProxy leader-routing tier, per location
    maxReplicas: 2
```

The `postgresML` block configures the bundled [`postgres-multi-location`](/template-catalog/templates/postgres-multi-location) cluster: one primary, asynchronous replicas in the other locations, an HAProxy leader-routing tier in each, and automatic promotion in a surviving location. Everything else the database tier can do — pooling, per-member addressing, emergency quorum recovery — is documented on that template's page, and its backup settings are covered under [Backing Up](#backing-up).

`primaryLocation` is a **preference**: if that location is slow to start, another one bootstraps the cluster and the primary lands there. Grafana has **no read/write splitting**, so every query, including every dashboard load, goes to the single primary — set `primaryLocation` where most of your users are, because every other location pays one cross-region round trip per query.

Leave the etcd compaction settings alone: a consensus store that fills its backend goes read-only and takes the database's failover with it. See [Compaction and Backend Growth](/template-catalog/templates/etcd-multi-location#compaction-and-backend-growth) for the mechanism and accepted value formats.

### Alerting HA Coordination

Rendered **only** when `alerting.highAvailability.enabled` is `true`; ignored entirely otherwise.

```yaml theme={null}
redisML:
  engine: redis # redis | valkey — chosen at INSTALL time; a data directory cannot be moved between engines
  valkeyImage: valkey/valkey:8.1.9 # used for both tiers when engine is valkey

  firewall:
    internalAllowType: same-gvc # options: same-gvc, same-org, workload-list
    workloads: []

  redis:
    passwordSecretName: "" # OPTIONAL opaque secret (encoding: plain) holding the Redis password
    image: redis:7.4
    replicasPerLocation: 1 # Redis members per location; 1 is plenty for alert coordination
    resources:
      cpu: 200m
      memory: 256Mi
    volumeset:
      initialCapacity: 10 # GiB per member (platform minimum); coordination data is tiny
  sentinel:
    passwordSecretName: "" # OPTIONAL opaque secret (encoding: plain) holding the Sentinel password
    image: redis:7.4
    resources:
      cpu: 200m
      memory: 256Mi
```

The block configures the bundled [`redis-multi-location`](/template-catalog/templates/redis-multi-location) subchart: one Sentinel and `replicasPerLocation` Redis members per location. `engine: valkey` runs both tiers on `valkeyImage` (the BSD-licensed fork of Redis 7.2, with `redis-server` / `redis-sentinel` compatibility symlinks), at which point `redis.image` and `sentinel.image` are inert.

#### Authenticating the Redis Tier

Authentication is off by default: enabling alerting HA is one flag and needs no extra secrets, and the tier is reachable only from inside the GVC. That GVC is yours and may already hold other workloads, so weigh the default accordingly — write access to an authless Redis is enough to **suppress your alert notifications**, because a hostile or buggy neighbour can claim another peer already sent them. Turn authentication on unless you control everything in the GVC.

Create either or both [opaque secrets](/guides/create-secret/opaque) with encoding `plain` **before installing**, then name them:

```bash theme={null}
printf '%s' "$(openssl rand -hex 32)" | cpln secret create-opaque --name REDIS_PASSWORD_SECRET_NAME --encoding plain -f -
printf '%s' "$(openssl rand -hex 32)" | cpln secret create-opaque --name SENTINEL_PASSWORD_SECRET_NAME --encoding plain -f -
```

```yaml theme={null}
redisML:
  redis:
    passwordSecretName: REDIS_PASSWORD_SECRET_NAME
  sentinel:
    passwordSecretName: SENTINEL_PASSWORD_SECRET_NAME
```

The two are independent — setting one without the other is valid. Grafana reads the **same secrets** the Redis tier reads to authenticate as a client, so the two sides cannot drift apart.

<Warning>
  **A password secret name that does not resolve stops the Grafana UI, not just alerting**, because Grafana mounts the same secret. `cpln helm install` still reports success while the Redis, Sentinel and Grafana UI tiers sit at 0 replicas — see the wedged-install warning under [Prerequisites](#prerequisites).
</Warning>

## Connecting

| What | Address | Notes |
| - | - | - |
| Grafana UI / API (public) | `status.canonicalEndpoint` of `RELEASE_NAME-grafana` | `cpln workload get RELEASE_NAME-grafana --gvc GVC_NAME -o yaml`. Georouted: a client is served by its nearest location |
| Grafana UI / API (internal) | `RELEASE_NAME-grafana.GVC_NAME.cpln.local:3000` | Served by an instance in the caller's own location |
| Alert evaluator (internal) | `RELEASE_NAME-grafana-alerting.GVC_NAME.cpln.local:3000` | Dedicated-evaluator mode only; answers only to callers inside `alerting.location` |
| App database | `RELEASE_NAME-postgres-proxy.GVC_NAME.cpln.local:5432` | Always the current primary, from any location |
| Alerting Sentinel (internal) | `replica-0.RELEASE_NAME-sentinel.LOCATION.GVC_NAME.cpln.local:26379` | Alerting HA only; master name `mymaster` |
| Admin login | `admin.user` plus the password in the admin password secret | `cpln secret reveal ADMIN_PASSWORD_SECRET_NAME -o yaml` |
| Health | `/api/health` on either Grafana address | Reports the Grafana version and the app-database status |

Verify an instance from inside the GVC — the Grafana image ships `curl`:

```bash theme={null}
cpln workload exec RELEASE_NAME-grafana --gvc GVC_NAME --container grafana -- curl -sS http://localhost:3000/api/health
```

Use the canonical `*.cpln.app` endpoint in browsers, not a per-location hostname: Grafana's root URL is the canonical endpoint, and on a per-location hostname the dashboard layout loads but panel data does not.

### First Run

Sign in on the public endpoint as `admin.user` with the password from the admin password secret. Grafana ships no dashboards and no datasources; connect your own in the UI or [as code](#datasources-as-code). Once you have signed in, change the password in the UI if you wish, then set `admin.applyPassword: false` and upgrade — Grafana only applies the secret when the account is first created, and dropping the reference lets you delete that secret.

## Operations

### Backing Up

Backups are off by default and cover the app database — the dashboards, users, alert rules and saved datasources that make up your Grafana. Enable them with `postgresML.backup.enabled: true`, choose a mode, and complete the storage setup for your provider **before** installing.

```yaml theme={null}
postgresML:
  backup:
    enabled: false
    mode: logical # logical or wal-g
    location: aws-us-east-1 # logical mode only: the ONE location the nightly job runs in
    resources:
      cpu: 100m
      memory: 512Mi # do not lower: the GCS path fails silently below this
    logical:
      image: ghcr.io/controlplane-com/backup-images/postgres-backup:17.1.0
      schedule: "0 2 * * *"
    walg:
      intervalSeconds: 21600
    provider: aws # options: aws, gcp, minio
    aws:
      bucket: my-grafana-bucket
      region: us-east-1
      cloudAccountName: my-s3-cloud-account
      policyName: my-grafana-backup-policy
      prefix: grafana/backups
    gcp:
      bucket: my-grafana-bucket
      cloudAccountName: my-gcs-cloud-account
      prefix: grafana/backups
    minio:
      endpoint: http://my-minio-workload:9000
      bucket: my-grafana-bucket
      credentialsSecretName: my-grafana-minio-credentials
      prefix: grafana/backups
```

`logical` runs a scheduled `pg_dumpall` as a cron workload in the one location named by `postgresML.backup.location`; `wal-g` archives continuously from whichever member is currently the primary. Both come from the [`postgres-multi-location`](/template-catalog/templates/postgres-multi-location#backing-up) subchart and were exercised in that template's testing, not through this one.

<Tabs>
  <Tab title="AWS S3">
    <Steps>
      <Step title="Create a bucket">
        Create an S3 bucket. Set `postgresML.backup.aws.bucket` and `postgresML.backup.aws.region` to match.
      </Step>

      <Step title="Set up a Cloud Account">
        If you do not have one, [create a Cloud Account](/guides/create-cloud-account) for the AWS account holding the bucket. Set `postgresML.backup.aws.cloudAccountName` to its name.
      </Step>

      <Step title="Create a bucket-scoped IAM policy">
        Create an IAM policy with the JSON below (replace `YOUR_BUCKET_NAME`), then set `postgresML.backup.aws.policyName` to the policy's name. No broad managed policy is required.

        ```json theme={null}
        {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Action": [
                        "s3:GetObject",
                        "s3:PutObject",
                        "s3:DeleteObject",
                        "s3:ListBucket",
                        "s3:GetObjectVersion",
                        "s3:DeleteObjectVersion"
                    ],
                    "Resource": [
                        "arn:aws:s3:::YOUR_BUCKET_NAME",
                        "arn:aws:s3:::YOUR_BUCKET_NAME/*"
                    ]
                }
            ]
        }
        ```
      </Step>

      <Step title="Choose a prefix">
        Set `postgresML.backup.aws.prefix` to the folder path within the bucket.
      </Step>
    </Steps>
  </Tab>

  <Tab title="Google Cloud Storage">
    <Steps>
      <Step title="Create a bucket">
        Create a GCS bucket. Set `postgresML.backup.gcp.bucket` to its name.
      </Step>

      <Step title="Set up a Cloud Account">
        If you do not have one, [create a Cloud Account](/guides/create-cloud-account) for the GCP project. Set `postgresML.backup.gcp.cloudAccountName` to its name.
      </Step>

      <Step title="Grant access">
        Add the **Storage Admin** (`roles/storage.admin`) role to the service account associated with the Cloud Account. The chart additionally binds `roles/storage.objectAdmin` on exactly the bucket named in `postgresML.backup.gcp.bucket`. Set `postgresML.backup.gcp.prefix` to the folder path.
      </Step>
    </Steps>
  </Tab>

  <Tab title="MinIO / S3-compatible">
    <Steps>
      <Step title="Create a bucket">
        Create the bucket on the server. Set `postgresML.backup.minio.bucket` to its name. No Cloud Account is needed.
      </Step>

      <Step title="Set the endpoint">
        Set `postgresML.backup.minio.endpoint` to the S3 API address including the port — for the [MinIO](/template-catalog/templates/minio) template in the same GVC, `http://WORKLOAD_NAME.GVC_NAME.cpln.local:9000`.

        **The endpoint must be reachable from every location.** In `wal-g` mode every database member reads from the archive, so a MinIO workload that runs in only one location leaves the members in the other locations unable to start — silently, because the primary stays healthy and writes keep succeeding. Run your S3-compatible endpoint in every location of the GVC, or use S3/GCS, which are global.
      </Step>

      <Step title="Create the credentials secret">
        Create a [dictionary secret](/guides/create-secret/dictionary), set `postgresML.backup.minio.credentialsSecretName` to its name, and set `postgresML.backup.minio.prefix` to the folder path:

        ```bash theme={null}
        cpln secret create-dictionary --name MINIO_CREDENTIALS_SECRET_NAME \
          --entry accessKey=MINIO_ACCESS_KEY \
          --entry secretKey=MINIO_SECRET_KEY
        ```
      </Step>
    </Steps>
  </Tab>
</Tabs>

### Restoring a Backup

A restore goes into the **database**, not into Grafana: everything that makes up your Grafana lives there, so restoring it moves dashboards, users, datasources, alert rules and alert state together. The restore procedures belong to the bundled database — follow [Restoring a Backup](/template-catalog/templates/postgres-multi-location#restoring-a-backup) on the `postgres-multi-location` page, substituting this release's proxy address, `RELEASE_NAME-postgres-proxy.GVC_NAME.cpln.local:5432`, and the credentials from your database credentials secret.

<Warning>
  **Neither restore path has been rehearsed through this template.** The logical path (`pg_dumpall` replayed through the proxy with `psql`) and the WAL-G path were verified in the `postgres-multi-location` template's own testing, and the WAL-G path there ends in a manual volume-set swap that the chart does not expose. Run a logical restore from a client workload in the same GVC that has both `psql` and your cloud CLI — neither the Grafana image nor the database image has both. Rehearse the procedure before you rely on it.
</Warning>

Two Grafana-specific rules apply to any restore:

* **Restore into a cluster that uses the same encryption key.** `admin.secretKeySecretName` decrypts the datasource credentials stored in the dump; a different key makes every saved datasource credential unreadable.
* **A zero-length backup object is a failed run, not a backup.** Check the object size before restoring from it — a real dump is kilobytes at minimum.

### Migrating From Version 1

Every release before 2.0.0 created its own GVC. 2.0.0 creates none and deploys into the GVC you install it into.

<Warning>
  **Never `helm upgrade` a 1.x release onto 2.0.0.** Once the chart stops declaring a GVC, Helm prunes the one the old release created — and deleting a GVC deletes every workload, volume set and identity inside it: both Grafana tiers, the whole database cluster and its data volumes, the consensus store and the Redis coordination tier, while the command reports success. The chart refuses to render if your values still carry the 1.x `global.gvc` key, but that guard **cannot** fire on an upgrade run with no values at all. The procedure below is the safety; the guard is only a backstop.
</Warning>

The refusal message usually names `etcd-multi-location` or `postgres-multi-location` rather than this chart, because every chart in the stack carries the same guard on the same key and a subchart's copy aborts the render first. The remedy is identical.

Install 2.0.0 as a **new release against an existing GVC**, restore the database into it, then remove the old release:

<Steps>
  <Step title="Back up the 1.x database">
    Enable `postgresML.backup.mode: logical` on the old release, or run `pg_dumpall` against the old release's `RELEASE_NAME-postgres-proxy` address from a client workload in that GVC. This one dump carries everything that makes up your Grafana.
  </Step>

  <Step title="Rewrite your values">
    Delete `global.gvc.name` and rename `global.gvc.locations` to `global.locations`. Every location you list must already exist in the GVC you are installing into.
  </Step>

  <Step title="Install 2.0.0 as a new release into an existing GVC">
    Not the GVC the 1.x release created — that GVC is still owned by the old Helm release and goes away when you uninstall it. Point `admin.secretKeySecretName` at the **same** secret the old release used.
  </Step>

  <Step title="Restore the dump and move your users">
    Follow [Restoring a Backup](#restoring-a-backup), then point your users at the new release's canonical endpoint.
  </Step>

  <Step title="Remove the old release">
    `cpln helm uninstall` the old release against the GVC you installed it into. That takes the GVC it created, and the old volume sets, with it.
  </Step>
</Steps>

Values that changed:

| 1.x | 2.0.0 |
| - | - |
| `global.gvc.name` | Removed — the GVC is wherever you install |
| `global.gvc.locations` | `global.locations` |
| `postgresML` pinned at 1.x | Pinned at **2.0.0**, which creates no GVC of its own |
| `redisML` pinned at 2.x | Pinned at **3.0.0**, which creates no GVC of its own and adds `redisML.engine` |

### Alert Evaluation

`alerting.enabled` is the parent switch. Setting it to `false` creates no evaluator workload and turns rule execution off on every instance: alert rules and contact points can still be created and viewed, and nothing ever evaluates them. While it is on, `alerting.highAvailability.enabled` picks one of two shapes:

| | `highAvailability.enabled: false` (default) | `highAvailability.enabled: true` |
| - | - | - |
| Who evaluates rules | The dedicated `RELEASE_NAME-grafana-alerting` workload — 1 replica, in `alerting.location` only | **Every UI instance**, in every location |
| Exactly-once notifications | There is only one evaluator | Redis-coordinated: the instances agree on one sender for each notification |
| Extra tier | None | 1 Sentinel and `redisML.redis.replicasPerLocation` Redis members **per location** |
| Losing `alerting.location` | Alert evaluation **stops** until you repoint `alerting.location` and upgrade | Evaluation **continues** in the surviving locations |
| Minimum locations | 2 | **3** — the chart refuses to render below that |
| Data-source query load | **1×** | **(locations × replicas)×** |
| `alerting.location` and `alerting.resources` | Apply | Ignored |

<Warning>
  **Turning HA on multiplies your data-source query load by the number of Grafana instances.** Every instance evaluates every rule against your Prometheus, Mimir or SQL server; Redis dedupes the **notification**, never the **query**. At the default 3 locations × 1 replica that is 3× the queries, permanently. If your data source is already the bottleneck, that is a worse trade than losing alert evaluation when a region dies — leave the knob off.
</Warning>

**Dedicated-evaluator mode (the default).** Rule execution is disabled on the UI tier and enabled on one workload with exactly one replica in `alerting.location`; nothing is elected at runtime, and no value of `replicas` can produce a second evaluator. Losing that replica self-heals — the platform reschedules it — but the replacement re-sends whatever is firing when it comes back, so a restart costs duplicate notifications rather than missed ones. Losing the **whole location** does not self-heal: evaluation stops until you run `helm upgrade` with `alerting.location` set to a surviving location, and the UI gives no sign of it. Moving `alerting.location` restarts only the evaluator and never runs two at once. Silences are **not expected to propagate** from the UI tier to the evaluator (not exercised in testing); to be safe, manage silences against the evaluator directly, from a workload running inside `alerting.location` — the address returns `503` from other locations:

```bash theme={null}
curl -u admin:PASSWORD http://RELEASE_NAME-grafana-alerting.GVC_NAME.cpln.local:3000/api/alertmanager/grafana/api/v2/silences
```

**Alerting HA.** No evaluator workload exists; every UI instance evaluates every rule and the stretched Redis tier makes sure exactly one of them sends each notification. Verified in testing by repeatedly destroying every replica in one location while the other two carried alerting exactly once; a whole-region outage has not been simulated, so survival of one follows from that result rather than from a timed failover. Two properties to design around:

* **Which location sends is not pinnable.** The sender moves between rollouts and between installs of the same chart, so do not build routing, filtering or egress-IP allowlisting on the assumption that a particular location sends.
* **An unhealthy Redis tier means duplicate notifications, never silence.** Each instance falls back to sending its own copy and re-converges on its own when Redis returns. Any `helm upgrade` with HA on — including one that changes nothing — also opens a brief duplicate window while the Redis tier is patched.

Turning HA on or off changes which workloads exist: enabling it deletes `RELEASE_NAME-grafana-alerting` and adds the Redis and Sentinel workloads in every location; disabling it does the reverse. Treat it as a planned change, not a toggle to flip during an incident.

### Scaling and Availability

* **`replicas`** scales the UI tier per location, in every location, with no alerting-related restriction. The only limit is the connection budget under [Grafana UI Tier](#grafana-ui-tier). `global.locations[].replicas` scales the **database** members instead.
* **Location loss.** With 3 or more locations the database fails over automatically and the endpoint keeps serving from the surviving locations; the location count table under [Prerequisites](#prerequisites) gives the arithmetic. What alerting does depends on the mode — see [Alert Evaluation](#alert-evaluation).
* **Every `helm upgrade` that touches the database tier is a planned outage.** The database members restart together rather than one at a time, and while the database is down every Grafana instance reports unhealthy and the endpoint returns `503`. The **first** upgrade after any install re-applies every tier once, even when nothing changed; later upgrades that only touch the Grafana workloads roll without downtime.
* **An upgrade that adds a new secret reference** (enabling `smtp` with a password, adding `datasources.credentialSecrets`, turning on backups) can pause the rollout for several minutes while Helm reports success, with `The identity ... is not allowed to reveal the secret ...` in the deployment status even though the grant is in place. It clears on its own; do not edit policies. Confirm the new version is live with `cpln workload get-deployments RELEASE_NAME-grafana --gvc GVC_NAME -o yaml` rather than trusting the Helm output.
* **Rotating a prerequisite secret does not restart anything.** After changing a secret's value, run `cpln workload force-redeployment RELEASE_NAME-grafana --gvc GVC_NAME` (and the same for `RELEASE_NAME-grafana-alerting` in dedicated-evaluator mode) so the instances pick it up. Never rotate the encryption key.
* **Removing a location** means removing it from `global.locations` **and** from the GVC. Never suspend a location on these workloads; use the values list.
* **Uninstall deletes every volume set the chart created** — the database, the consensus store and, with alerting HA on, Redis and Sentinel. The GVC is yours and is left alone; the prerequisite secrets you created survive. Enable backups if the data matters.

### Troubleshooting

<AccordionGroup>
  <Accordion title="A workload sits at 0 replicas after install while helm reported success">
    **Symptom:** `cpln helm install` succeeded, but `RELEASE_NAME-grafana` (and possibly the database or Redis tiers) show no containers and `cpln logs` returns nothing.

    **Cause:** A prerequisite secret named in your values does not exist — the admin password, the encryption key, the database credentials, or a `redisML.*.passwordSecretName`. The deployment pauses on the unresolved reference.

    **Fix:** Read the message naming the secret, then create it:

    ```bash theme={null}
    cpln workload get-deployments RELEASE_NAME-grafana --gvc GVC_NAME -o yaml
    ```

    The message reads `The secret <name> no longer exists. Workload updates are paused until the secret is added or the reference to the secret removed.` Once the secret exists the deployment recovers on its own, or run `cpln workload force-redeployment RELEASE_NAME-grafana --gvc GVC_NAME` to skip the wait.
  </Accordion>

  <Accordion title="WARNING locations declared in global.locations are not in GVC">
    **Symptom:** The Grafana logs carry `[grafana] WARNING: locations declared in global.locations are not in GVC '...'`, and a location you expected to serve traffic runs nothing.

    **Cause:** `global.locations` names a location the GVC does not have. The platform accepts and stores it without error, so nothing runs there and no deployment reports a failure. On a **fresh** install the database tier refuses to bootstrap in that state and crash-loops with a `FATAL:` line naming the location; on an install whose GVC lost a location later, the stack keeps serving and only this warning shows.

    **Fix:** Add the location to the GVC or remove it from `global.locations`, then upgrade. If the missing location is `alerting.location`, the `RELEASE_NAME-grafana-alerting` workload holds zero replicas everywhere and **no alert rule is evaluated** while every dashboard looks healthy — the warning says so explicitly. Check with:

    ```bash theme={null}
    cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-grafana"} |= "[grafana]"' --limit 50 --since 10m
    ```
  </Accordion>

  <Accordion title="Alert rules never fire but dashboards are healthy">
    **Symptom:** Rules exist and show no evaluation timestamp; no contact point ever receives anything.

    **Cause:** Either `alerting.enabled` is `false`, or you are in dedicated-evaluator mode and `alerting.location` is a location the GVC does not have (see the previous entry), or that location has been lost.

    **Fix:** Set `alerting.enabled: true`; repoint `alerting.location` to a surviving location listed in both your values and the GVC and upgrade; or enable `alerting.highAvailability.enabled` so evaluation runs in every location.
  </Accordion>

  <Accordion title="Every alert is delivered twice with alerting HA on">
    **Symptom:** Each firing alert produces one notification per Grafana instance.

    **Cause:** The coordination tier is unhealthy or unreachable: the `RELEASE_NAME-redis` or `RELEASE_NAME-sentinel` workloads are down, fewer than a majority of the configured locations exist in the GVC, or a `workload-list` firewall on the Redis tier does not include the Grafana workloads. Grafana logs `Failed to look up position, falling back to position 0`. It never goes silent, so the alert rules themselves are fine.

    **Fix:** Check those workloads with `cpln workload get-deployments RELEASE_NAME-redis --gvc GVC_NAME -o yaml`, restore the missing locations, or add the printed links to `redisML.firewall.workloads`. A brief duplicate window right after a `helm upgrade` is expected and clears on its own.
  </Accordion>

  <Accordion title="Failed to lock database on a cold start">
    **Symptom:** During a fresh install or a tier restart, one or more Grafana instances exit once with `Failed to lock database` and are restarted.

    **Cause:** Grafana runs its schema migrations under a non-blocking lock; an instance arriving while another holds it exits, and the next attempt succeeds against the migrated schema. The chart staggers the instances so this is rare at one replica per location; two replicas in the same location can still race.

    **Fix:** None needed. If it keeps happening for minutes, check that the database primary landed in `postgresML.primaryLocation` — migrations run across regions otherwise and take longer.
  </Accordion>

  <Accordion title="failed to send email unencrypted connection">
    **Symptom:** No alert emails arrive; the sending workload logs `failed to send email: unencrypted connection`.

    **Cause:** `smtp.user` is set and the relay does not offer STARTTLS or TLS. Grafana refuses to send credentials over an unencrypted connection and drops the notification.

    **Fix:** Use a relay that offers STARTTLS or TLS, or leave `smtp.user` empty to send unauthenticated against a plain relay.
  </Accordion>

  <Accordion title="A provisioned datasource reports upstream connect error connection timeout">
    **Symptom:** Panels on a datasource from `datasources.definitions` show only `upstream connect error ... connection timeout`.

    **Cause:** Almost always an unsubstituted placeholder in the datasource URL — the workload name or the GVC segment of `http://YOUR_WORKLOAD.YOUR_GVC.cpln.local:PORT` was left literal. A name that does not resolve times out rather than failing fast, so it looks like a network problem.

    **Fix:** Correct `datasources.definitions[].url` and upgrade.
  </Accordion>

  <Accordion title="Dashboards load but every panel is empty">
    **Symptom:** On a per-location hostname the UI and dashboard layout load, but no panel shows data and no error appears.

    **Cause:** Grafana's root URL is the canonical endpoint, and panel data requests from another hostname are rejected.

    **Fix:** Use the canonical `*.cpln.app` endpoint from `status.canonicalEndpoint`. It is georouted and already serves from the nearest location.
  </Accordion>

  <Accordion title="Render fails with the global.gvc values key was REMOVED">
    **Symptom:** `helm install` or `helm upgrade` aborts before touching anything, with a message naming `etcd-multi-location`, `postgres-multi-location` or this chart and the text `the global.gvc values key was REMOVED`.

    **Cause:** Your values file is from 1.x, which named a GVC under `global.gvc`.

    **Fix:** Do not retry as an upgrade. Follow [Migrating From Version 1](#migrating-from-version-1).
  </Accordion>

  <Accordion title="Render fails because the database connection budget is exceeded">
    **Symptom:** The install refuses with `the database connection budget is exceeded — N Grafana instances ... above the 80-connection ceiling`.

    **Cause:** `(replicas × locations + 1) × database.maxOpenConn` is above 80.

    **Fix:** Lower `database.maxOpenConn` or lower `replicas`; the message spells out the arithmetic.
  </Accordion>

  <Accordion title="Render fails because a subchart workload-list does not include the Grafana workloads">
    **Symptom:** The install refuses with `postgresML.internalAccess.type is 'workload-list' but ... does not include` or the `redisML.firewall.internalAllowType` equivalent.

    **Cause:** The database and Redis tiers keep their own firewall lists and do not add Grafana automatically; without the links Grafana could never reach its database or its alerting coordinator.

    **Fix:** Paste the links printed in the message into `postgresML.internalAccess.workloads` or `redisML.firewall.workloads`.
  </Accordion>
</AccordionGroup>

## Important Notes

* **Create the admin password, encryption key and database credentials secrets before installing.** Without them the deployment waits silently on secrets that do not exist while `helm install` reports success.
* **There is no upgrade path from 1.x.** A `helm upgrade` across the 2.0.0 boundary deletes the old GVC and everything in it. Follow [Migrating From Version 1](#migrating-from-version-1).
* **Every location in `global.locations` must already exist in the GVC you install into**, and `alerting.location` and `postgresML.primaryLocation` must be among them. GVC locations you do not list run nothing.
* **Never rotate or delete the encryption key.** Every stored datasource credential becomes unreadable in every location, and alert rules that query them fail.
* **Treat every `helm upgrade` that touches the database tier as a planned outage**, and expect the first upgrade after any install to re-apply every tier.
* **Alert evaluation stops if you lose `alerting.location`, and the UI will not show it.** Repoint the knob and upgrade, or enable alerting HA — after weighing its query-load cost.
* **Alerting HA multiplies data-source query load by (locations × replicas)** and is blocked below 3 locations on purpose.
* **With alerting HA on, an unhealthy Redis tier means duplicate notifications, never silence** — check the `RELEASE_NAME-redis` and `RELEASE_NAME-sentinel` workloads before suspecting your rules.
* **The alerting-HA Redis tier is unauthenticated by default.** Authenticate it unless you control everything in the GVC — see [Authenticating the Redis Tier](#authenticating-the-redis-tier).
* **Setting a subchart firewall to `workload-list` requires listing the Grafana workloads in it**; the chart refuses to render otherwise and prints the links.
* **Use the canonical `*.cpln.app` endpoint**, not a per-location hostname — panels stay empty otherwise.
* **Set `postgresML.primaryLocation` where most of your users are.** Grafana has no read/write splitting, so every other location pays a cross-region round trip per query.
* **After rotating a prerequisite secret, force a redeployment of the Grafana workloads**; nothing picks the new value up on its own.
* **Uninstall deletes the database volume sets.** Enable backups if the data matters; your prerequisite secrets survive.
* **This template ships Grafana OSS only** — Enterprise features such as fine-grained RBAC, reporting and query caching are not available.

## External References

<CardGroup cols={2}>
  <Card title="Grafana Documentation" icon="book" href="https://grafana.com/docs/grafana/latest/">
    Official Grafana documentation
  </Card>

  <Card title="Grafana Alerting" icon="bell" href="https://grafana.com/docs/grafana/latest/alerting/">
    Alert rules, contact points, notification policies, and silences
  </Card>

  <Card title="Alerting High Availability" icon="sliders" href="https://grafana.com/docs/grafana/latest/alerting/set-up/configure-high-availability/">
    How Grafana coordinates alert evaluation across instances
  </Card>

  <Card title="Provisioning Datasources" icon="database" href="https://grafana.com/docs/grafana/latest/administration/provisioning/">
    Reference for the datasource provisioning entries used by this template
  </Card>

  <Card title="Configuration Reference" icon="gear" href="https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/">
    Every Grafana setting and its environment-variable name
  </Card>
</CardGroup>