Grafana Multi-Location

Overview

Grafana Multi-Location deploys one logical Grafana whose UI/API instances run in every location you configure, behind a single georouted *.cpln.app endpoint. Every dashboard, user, org, session, alert rule and saved datasource lives in a bundled postgres-multi-location cluster stretched across the same locations, so Grafana itself holds no state — there is no volume, no session affinity and nothing to hand over when an instance is replaced. Alerting comes in two shapes, chosen with alerting.highAvailability.enabled: a single dedicated evaluator pinned to one location (the default), or every instance evaluating with a stretched Redis tier making sure exactly one of them sends each notification. See Alert Evaluation before choosing.

This template deploys into a GVC you already have — it does not create, provision or manage one. That GVC must have at least 2 locations (3 with alerting HA enabled), and every location you list in global.locations must already be one of them. GVC locations you do not list are left alone: nothing from this release runs there. For a Grafana inside a single location, use the Grafana template instead.

What Gets Created

Resource	Name	Purpose	Optional
Workload	`RELEASE_NAME-grafana`	Grafana UI/API tier (standard, `replicas` instances per location, HTTP port `3000`, public by default)	—
Workload	`RELEASE_NAME-grafana-alerting`	Dedicated alert evaluator (standard, exactly 1 replica, in `alerting.location` only, never public)	Default; not created when `alerting.highAvailability.enabled: true` or `alerting.enabled: false`
Workload	`RELEASE_NAME-postgres`	Patroni PostgreSQL app database (stateful, `global.locations[].replicas` members per location)	—
Workload	`RELEASE_NAME-postgres-proxy`	HAProxy leader-routing tier (standard, per location, always points at the current primary)	—
Workload	`RELEASE_NAME-etcd`	Consensus store for the database (stateful, exactly 1 member per location)	—
Workload	`RELEASE_NAME-postgres-backup`	Scheduled logical database backup (cron, one location)	`postgresML.backup.enabled: true` with `mode: logical`
Workload	`RELEASE_NAME-redis`, `RELEASE_NAME-sentinel`	Alerting coordination tier (stateful, one Sentinel and `redisML.redis.replicasPerLocation` Redis members per location)	`alerting.highAvailability.enabled: true`
Volume set	`RELEASE_NAME-postgres-vs`, `RELEASE_NAME-etcd-vs`	Database and consensus-store data (10 GiB per member by default)	—
Volume set	`RELEASE_NAME-redis-vs`, `RELEASE_NAME-sentinel-vs`	Coordination-tier data	`alerting.highAvailability.enabled: true`
Secret	`RELEASE_NAME-grafana-datasources`	Rendered datasource provisioning file, mounted by every Grafana workload	`datasources.definitions` non-empty
Secret	`RELEASE_NAME-postgres-startup`, `RELEASE_NAME-postgres-proxy-startup`, `RELEASE_NAME-etcd-startup`	Startup scripts of the database tier	—
Secret	`RELEASE_NAME-redis-config`, `RELEASE_NAME-sentinel-config`	Coordination-tier configuration	`alerting.highAvailability.enabled: true`
Identity	`RELEASE_NAME-grafana-identity`	Shared by both Grafana workloads	—
Policy	`RELEASE_NAME-grafana-policy`	`reveal` on exactly the secrets the Grafana workloads mount	—
Policy	`RELEASE_NAME-grafana-gvc-policy`	`view` on the one GVC this release installs into, so each instance can check the GVC’s location list at boot	—
Identity / Policy	`RELEASE_NAME-postgres-identity`, `RELEASE_NAME-postgres-policy`, `RELEASE_NAME-postgres-gvc-policy`, `RELEASE_NAME-etcd-identity`, `RELEASE_NAME-etcd-policy`, `RELEASE_NAME-etcd-gvc-policy`	The database tier’s own identities and policies	—
Identity / Policy	`RELEASE_NAME-redis-identity`, `RELEASE_NAME-redis-policy`, `RELEASE_NAME-redis-gvc-policy`, `RELEASE_NAME-sentinel-identity`, `RELEASE_NAME-sentinel-policy`	The coordination tier’s own identities and policies	`alerting.highAvailability.enabled: true`

The admin password, the encryption key and the database credentials are prerequisite secrets you create yourself; the chart references them by name and never creates, modifies or deletes them.

Prerequisites

Have a GVC with at least 2 locations

This template creates no GVC. Read the locations of the GVC you will install into and make every entry of global.locations one of them:

cpln gvc get GVC_NAME -o yaml

Compare spec.staticPlacement.locationLinks with your values. The database commits a write only when a majority of its members agree, and it runs one member per location, so the location count decides what survives:

Locations	Location losses survived	What a user sees when one location is lost
2	0	Dashboards may still render, but logins, saves and alert-state writes fail until the survivor is promoted by hand.
3	1	Automatic database failover; the endpoint keeps serving from the two surviving locations.
5	2	Survives losing two locations.

With N locations you survive floor((N-1)/2) losses, so an even count buys nothing over the odd count below it. alerting.highAvailability.enabled: true requires at least 3 locations, and the chart refuses to render below that.

Create the first-boot admin password

An opaque secret with encoding plain. Grafana reads it only when the admin account is first created:

printf '%s' "$(openssl rand -hex 24)" | cpln secret create-opaque --name ADMIN_PASSWORD_SECRET_NAME --encoding plain -f -

Set admin.passwordSecretName to this name.

Create the encryption key

An opaque secret with encoding plain. Every instance in every location reads it on every boot to decrypt datasource credentials stored in the shared database — back it up outside Control Plane and never rotate it:

printf '%s' "$(openssl rand -hex 32)" | cpln secret create-opaque --name SECRET_KEY_SECRET_NAME --encoding plain -f -

Set admin.secretKeySecretName to this name.

Create the app database credentials

A dictionary secret holding exactly username, password and database. Use plain identifiers for username and database — they are used unquoted when the database is created:

cpln secret create-dictionary --name DB_CREDENTIALS_SECRET_NAME \
  --entry username=grafana \
  --entry password="$(openssl rand -hex 24)" \
  --entry database=grafana

Set postgresML.postgres.credentialsSecretName to this name.

A missing prerequisite secret wedges the install rather than failing it. cpln helm install reports success while the affected workloads sit at zero replicas with no containers and therefore no logs. The only place the missing secret is named is status.versions[].message:

cpln workload get-deployments RELEASE_NAME-grafana --gvc GVC_NAME -o yaml

Create the missing secret and the deployment recovers on its own after several minutes, or run cpln workload force-redeployment RELEASE_NAME-grafana --gvc GVC_NAME to skip the wait.

Optional features each need something created before you install:

Feature	What you must create first
Credentialed provisioned datasources	One dictionary secret per entry in `datasources.credentialSecrets` — see Datasources as Code
Authenticated SMTP	An opaque secret with encoding `plain` holding the SMTP password, named in `smtp.passwordSecretName`
Authenticated alerting-HA Redis tier	One opaque secret with encoding `plain` per password — see Authenticating the Redis Tier. Alerting HA itself needs no extra secrets
Database backups	A bucket and access setup on AWS S3, Google Cloud Storage or an S3-compatible server — see Backing Up

Installation

Once the three secrets exist, install the template into the GVC with the required secret names:

cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/grafana-multi-location \
  --version 2.0.0 --gvc GVC_NAME \
  --set admin.passwordSecretName=ADMIN_PASSWORD_SECRET_NAME \
  --set admin.secretKeySecretName=SECRET_KEY_SECRET_NAME \
  --set postgresML.postgres.credentialsSecretName=DB_CREDENTIALS_SECRET_NAME

The default global.locations list is aws-us-east-1, aws-eu-central-1 and aws-us-west-2. If your GVC’s locations differ, supply your own list in a values file (--values) alongside the command above, and keep alerting.location and postgresML.primaryLocation pointing at one of the locations you list.

UI

Browse, install, and manage templates visually

CLI

Manage templates from your terminal

Terraform

Declare templates in your Terraform configurations

Pulumi

Declare templates in your Pulumi programs

Configuration

Locations

# Every location listed here MUST already exist in the GVC you install into.
# Extra locations in the GVC are fine — nothing here runs in them.
# Minimum 2 locations, and minimum 3 with alerting HA enabled.
# `replicas` here sets the POSTGRES members in that location, and nothing else:
#   etcd     — always exactly 1 per location, not configurable.
#   Grafana  — the top-level `replicas` below, applied to every location.
#   Redis    — redisML.redis.replicasPerLocation (alerting HA only).
global:
  locations:
    - name: aws-us-east-1
      replicas: 1
    - name: aws-eu-central-1
      replicas: 1
    - name: aws-us-west-2
      replicas: 1

The list lives under global so the bundled database, its consensus store and the optional Redis tier all receive the same list — there is never a second list to maintain. Removing a location from the list is the supported way to shrink the deployment; replicas must stay at least 1. Both Grafana tiers read the GVC at boot and log a [grafana] WARNING: line naming any location in global.locations that the GVC does not have. See Troubleshooting for what that means for alerting.

Grafana UI Tier

image: grafana/grafana:13.1.3

replicas: 1 # Grafana UI instances PER LOCATION

resources:
  maxCpu: 1000m
  maxMemory: 1Gi
  minCpu: 500m
  minMemory: 512Mi

database:
  maxOpenConn: 10 # per instance; see the connection budget below

(replicas × locations + 1) × database.maxOpenConn must stay at 80 or less; the + 1 is the dedicated alert evaluator and is dropped when there is none (alerting HA on, or alerting.enabled: false). The chart enforces this at render time and refuses to install with the arithmetic spelled out, so raising replicas past that point means lowering database.maxOpenConn.

image must provide /bin/bash — the chart wraps Grafana’s own entrypoint in a boot script; the default official image does. Signup, anonymous access and upstream analytics are disabled by the template and are not configurable. With alerting HA on, rule evaluation runs on this tier, so a large rule set may need resources.maxCpu raised.

Alerting

alerting:
  enabled: true # false = rules can be created and viewed, nothing ever evaluates them
  highAvailability:
    enabled: false # true = every UI instance evaluates, Redis coordinates the send; needs 3+ locations
  # Dedicated-evaluator mode ONLY — where the one evaluator runs. Must be one of
  # global.locations. IGNORED when highAvailability.enabled is true.
  location: aws-us-east-1
  # Dedicated-evaluator mode ONLY — in HA mode the UI tier's `resources` apply.
  resources:
    maxCpu: 1000m
    maxMemory: 1Gi
    minCpu: 500m
    minMemory: 512Mi

Read Alert Evaluation before enabling highAvailability — it multiplies your data-source query load by the number of Grafana instances. alerting.enabled: false combined with highAvailability.enabled: true fails at render time.

Admin Credentials and Encryption Key

admin:
  user: admin # initial admin login name (not sensitive)
  applyPassword: true # set false after your first login to stop referencing the password secret
  passwordSecretName: my-grafana-admin-password # opaque secret holding the first-boot admin password
  secretKeySecretName: my-grafana-secret-key # opaque secret holding the at-rest encryption key

Both are opaque secrets you create before installing (see Prerequisites); neither value ever passes through Helm. Their lifecycles are opposites:

	`admin.passwordSecretName`	`admin.secretKeySecretName`
What it holds	The admin login password	The key encrypting stored datasource credentials
When Grafana reads it	Only when the admin account is first created	On every boot, on every instance
Can you stop referencing it?	Yes — set `admin.applyPassword: false` once you have logged in	No
Can you rotate or delete it?	Change the password in the UI, then set `applyPassword: false` and delete the secret	Never — every stored datasource credential becomes unreadable in every location

Datasources as Code

datasources:
  definitions: []
  # - name: Prometheus
  #   type: prometheus
  #   access: proxy
  #   # Substitute BOTH parts: the workload name AND the GVC.
  #   url: http://YOUR_WORKLOAD.YOUR_GVC.cpln.local:9095
  #   isDefault: true
  credentialSecrets: []
  # - name: my-grafana-ds-credentials
  #   keys: [PG_PASSWORD]

definitions are standard Grafana datasource provisioning entries, rendered verbatim into a file that every Grafana instance applies at boot. The file is plaintext, so credentials never go into it: create a dictionary secret before installing, list it under credentialSecrets, and write $KEY in the definition. Each listed key becomes an environment variable the chart grants the identity reveal on, and Grafana interpolates it when it loads the file:

cpln secret create-dictionary --name DATASOURCE_CREDENTIALS_SECRET_NAME \
  --entry PG_PASSWORD='YOUR-DATASOURCE-PASSWORD'

datasources:
  definitions:
    - name: AppDB
      type: postgres
      url: my-db-host:5432
      user: grafana_reader
      jsonData: { database: appdb, sslmode: disable }
      secureJsonData:
        password: $PG_PASSWORD
  credentialSecrets:
    - name: DATASOURCE_CREDENTIALS_SECRET_NAME
      keys: [PG_PASSWORD]

Provisioned datasources are read-only in the UI — change the values and upgrade to change one.

SMTP

smtp:
  enabled: false
  host: smtp.example.com:587 # host:port
  user: "" # empty = unauthenticated SMTP; if set, the relay MUST offer STARTTLS or TLS
  passwordSecretName: "" # opaque secret (encoding: plain) with the SMTP password; create BEFORE install
  fromAddress: grafana@example.com
  fromName: Grafana

Whichever instance evaluates is what sends: the dedicated evaluator by default, or the coordinating UI instance with alerting HA on. smtp.passwordSecretName is required whenever smtp.user is set:

printf '%s' 'YOUR-SMTP-PASSWORD' | cpln secret create-opaque --name SMTP_PASSWORD_SECRET_NAME --encoding plain -f -

Authenticated SMTP requires a relay that offers STARTTLS or TLS. Grafana refuses to send credentials over an unencrypted connection, fails with failed to send email: unencrypted connection, and every notification is lost — the only signal is a log line in the sending workload. Hosted relays (SES, SendGrid, Mailgun, Microsoft 365, Gmail) are fine; a plain in-GVC relay is not. Leave smtp.user empty for such relays.

Access

publicAccess:
  enabled: true # UI on the canonical *.cpln.app HTTPS endpoint

internalAccess:
  type: same-gvc # options: none, same-gvc, same-org, workload-list
  # Only used when type is workload-list. This release's OWN workloads are added
  # automatically — list your clients here, not this release's workloads.
  workloads: []
  # workloads:
  #   - //gvc/GVC_NAME/workload/WORKLOAD_NAME

publicAccess applies to the UI tier only; with it on, links in alert notifications point at the public endpoint. The dedicated alert evaluator is never reachable from the internet and honours internalAccess only. With alerting HA on there is no evaluator, and the Redis tier is reachable from inside the GVC only.

The bundled database and Redis tiers keep their own firewall lists, and the Grafana workloads are not added to them automatically. Setting postgresML.internalAccess.type or redisML.firewall.internalAllowType to workload-list without listing the Grafana workloads would cut Grafana off from its database or its alerting coordinator, so the chart refuses to render in that state and prints the exact links to paste into the subchart’s workloads list.

Allow several minutes after changing an access knob before concluding it did not apply.

App Database

postgresML:
  etcd:
    tuning:
      autoCompactionMode: periodic # periodic (retention is a duration) or revision (a revision count)
      autoCompactionRetention: 1h # periodic needs an explicit unit (1h, 30m, 24h)
      quotaBackendBytes: 0 # backend size limit in bytes; 0 = etcd's own default of 2 GiB

  postgres:
    credentialsSecretName: my-grafana-db-credentials # REQUIRED prerequisite dictionary secret

  internalAccess:
    type: same-gvc # options: same-gvc, same-org, workload-list
    workloads: []

  # Preferred location for the database primary. Keep it aligned with
  # alerting.location so the hot path has no cross-region hop.
  primaryLocation: aws-us-east-1

  resources:
    minCpu: 500m
    minMemory: 1Gi
    maxCpu: 1
    maxMemory: 2Gi

  volumeset:
    capacity: 10 # initial capacity in GiB per member (minimum is 10)

  proxy:
    minReplicas: 2 # HAProxy leader-routing tier, per location
    maxReplicas: 2

The postgresML block configures the bundled postgres-multi-location cluster: one primary, asynchronous replicas in the other locations, an HAProxy leader-routing tier in each, and automatic promotion in a surviving location. Everything else the database tier can do — pooling, per-member addressing, emergency quorum recovery — is documented on that template’s page, and its backup settings are covered under Backing Up. primaryLocation is a preference: if that location is slow to start, another one bootstraps the cluster and the primary lands there. Grafana has no read/write splitting, so every query, including every dashboard load, goes to the single primary — set primaryLocation where most of your users are, because every other location pays one cross-region round trip per query. Leave the etcd compaction settings alone: a consensus store that fills its backend goes read-only and takes the database’s failover with it. See Compaction and Backend Growth for the mechanism and accepted value formats.

Alerting HA Coordination

Rendered only when alerting.highAvailability.enabled is true; ignored entirely otherwise.

redisML:
  engine: redis # redis | valkey — chosen at INSTALL time; a data directory cannot be moved between engines
  valkeyImage: valkey/valkey:8.1.9 # used for both tiers when engine is valkey

  firewall:
    internalAllowType: same-gvc # options: same-gvc, same-org, workload-list
    workloads: []

  redis:
    passwordSecretName: "" # OPTIONAL opaque secret (encoding: plain) holding the Redis password
    image: redis:7.4
    replicasPerLocation: 1 # Redis members per location; 1 is plenty for alert coordination
    resources:
      cpu: 200m
      memory: 256Mi
    volumeset:
      initialCapacity: 10 # GiB per member (platform minimum); coordination data is tiny
  sentinel:
    passwordSecretName: "" # OPTIONAL opaque secret (encoding: plain) holding the Sentinel password
    image: redis:7.4
    resources:
      cpu: 200m
      memory: 256Mi

The block configures the bundled redis-multi-location subchart: one Sentinel and replicasPerLocation Redis members per location. engine: valkey runs both tiers on valkeyImage (the BSD-licensed fork of Redis 7.2, with redis-server / redis-sentinel compatibility symlinks), at which point redis.image and sentinel.image are inert.

Authenticating the Redis Tier

Authentication is off by default: enabling alerting HA is one flag and needs no extra secrets, and the tier is reachable only from inside the GVC. That GVC is yours and may already hold other workloads, so weigh the default accordingly — write access to an authless Redis is enough to suppress your alert notifications, because a hostile or buggy neighbour can claim another peer already sent them. Turn authentication on unless you control everything in the GVC. Create either or both opaque secrets with encoding plain before installing, then name them:

printf '%s' "$(openssl rand -hex 32)" | cpln secret create-opaque --name REDIS_PASSWORD_SECRET_NAME --encoding plain -f -
printf '%s' "$(openssl rand -hex 32)" | cpln secret create-opaque --name SENTINEL_PASSWORD_SECRET_NAME --encoding plain -f -

redisML:
  redis:
    passwordSecretName: REDIS_PASSWORD_SECRET_NAME
  sentinel:
    passwordSecretName: SENTINEL_PASSWORD_SECRET_NAME

The two are independent — setting one without the other is valid. Grafana reads the same secrets the Redis tier reads to authenticate as a client, so the two sides cannot drift apart.

A password secret name that does not resolve stops the Grafana UI, not just alerting, because Grafana mounts the same secret. cpln helm install still reports success while the Redis, Sentinel and Grafana UI tiers sit at 0 replicas — see the wedged-install warning under Prerequisites.

Connecting

What	Address	Notes
Grafana UI / API (public)	`status.canonicalEndpoint` of `RELEASE_NAME-grafana`	`cpln workload get RELEASE_NAME-grafana --gvc GVC_NAME -o yaml`. Georouted: a client is served by its nearest location
Grafana UI / API (internal)	`RELEASE_NAME-grafana.GVC_NAME.cpln.local:3000`	Served by an instance in the caller’s own location
Alert evaluator (internal)	`RELEASE_NAME-grafana-alerting.GVC_NAME.cpln.local:3000`	Dedicated-evaluator mode only; answers only to callers inside `alerting.location`
App database	`RELEASE_NAME-postgres-proxy.GVC_NAME.cpln.local:5432`	Always the current primary, from any location
Alerting Sentinel (internal)	`replica-0.RELEASE_NAME-sentinel.LOCATION.GVC_NAME.cpln.local:26379`	Alerting HA only; master name `mymaster`
Admin login	`admin.user` plus the password in the admin password secret	`cpln secret reveal ADMIN_PASSWORD_SECRET_NAME -o yaml`
Health	`/api/health` on either Grafana address	Reports the Grafana version and the app-database status

Verify an instance from inside the GVC — the Grafana image ships curl:

cpln workload exec RELEASE_NAME-grafana --gvc GVC_NAME --container grafana -- curl -sS http://localhost:3000/api/health

Use the canonical *.cpln.app endpoint in browsers, not a per-location hostname: Grafana’s root URL is the canonical endpoint, and on a per-location hostname the dashboard layout loads but panel data does not.

First Run

Sign in on the public endpoint as admin.user with the password from the admin password secret. Grafana ships no dashboards and no datasources; connect your own in the UI or as code. Once you have signed in, change the password in the UI if you wish, then set admin.applyPassword: false and upgrade — Grafana only applies the secret when the account is first created, and dropping the reference lets you delete that secret.

Operations

Backing Up

Backups are off by default and cover the app database — the dashboards, users, alert rules and saved datasources that make up your Grafana. Enable them with postgresML.backup.enabled: true, choose a mode, and complete the storage setup for your provider before installing.

postgresML:
  backup:
    enabled: false
    mode: logical # logical or wal-g
    location: aws-us-east-1 # logical mode only: the ONE location the nightly job runs in
    resources:
      cpu: 100m
      memory: 512Mi # do not lower: the GCS path fails silently below this
    logical:
      image: ghcr.io/controlplane-com/backup-images/postgres-backup:17.1.0
      schedule: "0 2 * * *"
    walg:
      intervalSeconds: 21600
    provider: aws # options: aws, gcp, minio
    aws:
      bucket: my-grafana-bucket
      region: us-east-1
      cloudAccountName: my-s3-cloud-account
      policyName: my-grafana-backup-policy
      prefix: grafana/backups
    gcp:
      bucket: my-grafana-bucket
      cloudAccountName: my-gcs-cloud-account
      prefix: grafana/backups
    minio:
      endpoint: http://my-minio-workload:9000
      bucket: my-grafana-bucket
      credentialsSecretName: my-grafana-minio-credentials
      prefix: grafana/backups

logical runs a scheduled pg_dumpall as a cron workload in the one location named by postgresML.backup.location; wal-g archives continuously from whichever member is currently the primary. Both come from the postgres-multi-location subchart and were exercised in that template’s testing, not through this one.

AWS S3
Google Cloud Storage
MinIO / S3-compatible

Create a bucket

Create an S3 bucket. Set postgresML.backup.aws.bucket and postgresML.backup.aws.region to match.

Set up a Cloud Account

If you do not have one, create a Cloud Account for the AWS account holding the bucket. Set postgresML.backup.aws.cloudAccountName to its name.

Create a bucket-scoped IAM policy

Create an IAM policy with the JSON below (replace YOUR_BUCKET_NAME), then set postgresML.backup.aws.policyName to the policy’s name. No broad managed policy is required.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:ListBucket",
                "s3:GetObjectVersion",
                "s3:DeleteObjectVersion"
            ],
            "Resource": [
                "arn:aws:s3:::YOUR_BUCKET_NAME",
                "arn:aws:s3:::YOUR_BUCKET_NAME/*"
            ]
        }
    ]
}

Choose a prefix

Set postgresML.backup.aws.prefix to the folder path within the bucket.

Create a bucket

Create a GCS bucket. Set postgresML.backup.gcp.bucket to its name.

Set up a Cloud Account

If you do not have one, create a Cloud Account for the GCP project. Set postgresML.backup.gcp.cloudAccountName to its name.

Grant access

Add the Storage Admin (roles/storage.admin) role to the service account associated with the Cloud Account. The chart additionally binds roles/storage.objectAdmin on exactly the bucket named in postgresML.backup.gcp.bucket. Set postgresML.backup.gcp.prefix to the folder path.

Create a bucket

Create the bucket on the server. Set postgresML.backup.minio.bucket to its name. No Cloud Account is needed.

Set the endpoint

Set postgresML.backup.minio.endpoint to the S3 API address including the port — for the MinIO template in the same GVC, http://WORKLOAD_NAME.GVC_NAME.cpln.local:9000.The endpoint must be reachable from every location. In wal-g mode every database member reads from the archive, so a MinIO workload that runs in only one location leaves the members in the other locations unable to start — silently, because the primary stays healthy and writes keep succeeding. Run your S3-compatible endpoint in every location of the GVC, or use S3/GCS, which are global.

Create the credentials secret

Create a dictionary secret, set postgresML.backup.minio.credentialsSecretName to its name, and set postgresML.backup.minio.prefix to the folder path:

cpln secret create-dictionary --name MINIO_CREDENTIALS_SECRET_NAME \
  --entry accessKey=MINIO_ACCESS_KEY \
  --entry secretKey=MINIO_SECRET_KEY

Restoring a Backup

A restore goes into the database, not into Grafana: everything that makes up your Grafana lives there, so restoring it moves dashboards, users, datasources, alert rules and alert state together. The restore procedures belong to the bundled database — follow Restoring a Backup on the postgres-multi-location page, substituting this release’s proxy address, RELEASE_NAME-postgres-proxy.GVC_NAME.cpln.local:5432, and the credentials from your database credentials secret.

Neither restore path has been rehearsed through this template. The logical path (pg_dumpall replayed through the proxy with psql) and the WAL-G path were verified in the postgres-multi-location template’s own testing, and the WAL-G path there ends in a manual volume-set swap that the chart does not expose. Run a logical restore from a client workload in the same GVC that has both psql and your cloud CLI — neither the Grafana image nor the database image has both. Rehearse the procedure before you rely on it.

Two Grafana-specific rules apply to any restore:

Restore into a cluster that uses the same encryption key. admin.secretKeySecretName decrypts the datasource credentials stored in the dump; a different key makes every saved datasource credential unreadable.
A zero-length backup object is a failed run, not a backup. Check the object size before restoring from it — a real dump is kilobytes at minimum.

Migrating From Version 1

Every release before 2.0.0 created its own GVC. 2.0.0 creates none and deploys into the GVC you install it into.

Never helm upgrade a 1.x release onto 2.0.0. Once the chart stops declaring a GVC, Helm prunes the one the old release created — and deleting a GVC deletes every workload, volume set and identity inside it: both Grafana tiers, the whole database cluster and its data volumes, the consensus store and the Redis coordination tier, while the command reports success. The chart refuses to render if your values still carry the 1.x global.gvc key, but that guard cannot fire on an upgrade run with no values at all. The procedure below is the safety; the guard is only a backstop.

The refusal message usually names etcd-multi-location or postgres-multi-location rather than this chart, because every chart in the stack carries the same guard on the same key and a subchart’s copy aborts the render first. The remedy is identical. Install 2.0.0 as a new release against an existing GVC, restore the database into it, then remove the old release:

Back up the 1.x database

Enable postgresML.backup.mode: logical on the old release, or run pg_dumpall against the old release’s RELEASE_NAME-postgres-proxy address from a client workload in that GVC. This one dump carries everything that makes up your Grafana.

Rewrite your values

Delete global.gvc.name and rename global.gvc.locations to global.locations. Every location you list must already exist in the GVC you are installing into.

Install 2.0.0 as a new release into an existing GVC

Not the GVC the 1.x release created — that GVC is still owned by the old Helm release and goes away when you uninstall it. Point admin.secretKeySecretName at the same secret the old release used.

Restore the dump and move your users

Follow Restoring a Backup, then point your users at the new release’s canonical endpoint.

Remove the old release

cpln helm uninstall the old release against the GVC you installed it into. That takes the GVC it created, and the old volume sets, with it.

Values that changed:

1.x	2.0.0
`global.gvc.name`	Removed — the GVC is wherever you install
`global.gvc.locations`	`global.locations`
`postgresML` pinned at 1.x	Pinned at 2.0.0, which creates no GVC of its own
`redisML` pinned at 2.x	Pinned at 3.0.0, which creates no GVC of its own and adds `redisML.engine`

Alert Evaluation

alerting.enabled is the parent switch. Setting it to false creates no evaluator workload and turns rule execution off on every instance: alert rules and contact points can still be created and viewed, and nothing ever evaluates them. While it is on, alerting.highAvailability.enabled picks one of two shapes:

	`highAvailability.enabled: false` (default)	`highAvailability.enabled: true`
Who evaluates rules	The dedicated `RELEASE_NAME-grafana-alerting` workload — 1 replica, in `alerting.location` only	Every UI instance, in every location
Exactly-once notifications	There is only one evaluator	Redis-coordinated: the instances agree on one sender for each notification
Extra tier	None	1 Sentinel and `redisML.redis.replicasPerLocation` Redis members per location
Losing `alerting.location`	Alert evaluation stops until you repoint `alerting.location` and upgrade	Evaluation continues in the surviving locations
Minimum locations	2	3 — the chart refuses to render below that
Data-source query load	1×	(locations × replicas)×
`alerting.location` and `alerting.resources`	Apply	Ignored

Turning HA on multiplies your data-source query load by the number of Grafana instances. Every instance evaluates every rule against your Prometheus, Mimir or SQL server; Redis dedupes the notification, never the query. At the default 3 locations × 1 replica that is 3× the queries, permanently. If your data source is already the bottleneck, that is a worse trade than losing alert evaluation when a region dies — leave the knob off.

Dedicated-evaluator mode (the default). Rule execution is disabled on the UI tier and enabled on one workload with exactly one replica in alerting.location; nothing is elected at runtime, and no value of replicas can produce a second evaluator. Losing that replica self-heals — the platform reschedules it — but the replacement re-sends whatever is firing when it comes back, so a restart costs duplicate notifications rather than missed ones. Losing the whole location does not self-heal: evaluation stops until you run helm upgrade with alerting.location set to a surviving location, and the UI gives no sign of it. Moving alerting.location restarts only the evaluator and never runs two at once. Silences are not expected to propagate from the UI tier to the evaluator (not exercised in testing); to be safe, manage silences against the evaluator directly, from a workload running inside alerting.location — the address returns 503 from other locations:

curl -u admin:PASSWORD http://RELEASE_NAME-grafana-alerting.GVC_NAME.cpln.local:3000/api/alertmanager/grafana/api/v2/silences

Alerting HA. No evaluator workload exists; every UI instance evaluates every rule and the stretched Redis tier makes sure exactly one of them sends each notification. Verified in testing by repeatedly destroying every replica in one location while the other two carried alerting exactly once; a whole-region outage has not been simulated, so survival of one follows from that result rather than from a timed failover. Two properties to design around:

Which location sends is not pinnable. The sender moves between rollouts and between installs of the same chart, so do not build routing, filtering or egress-IP allowlisting on the assumption that a particular location sends.
An unhealthy Redis tier means duplicate notifications, never silence. Each instance falls back to sending its own copy and re-converges on its own when Redis returns. Any helm upgrade with HA on — including one that changes nothing — also opens a brief duplicate window while the Redis tier is patched.

Turning HA on or off changes which workloads exist: enabling it deletes RELEASE_NAME-grafana-alerting and adds the Redis and Sentinel workloads in every location; disabling it does the reverse. Treat it as a planned change, not a toggle to flip during an incident.

Scaling and Availability

replicas scales the UI tier per location, in every location, with no alerting-related restriction. The only limit is the connection budget under Grafana UI Tier. global.locations[].replicas scales the database members instead.
Location loss. With 3 or more locations the database fails over automatically and the endpoint keeps serving from the surviving locations; the location count table under Prerequisites gives the arithmetic. What alerting does depends on the mode — see Alert Evaluation.
Every helm upgrade that touches the database tier is a planned outage. The database members restart together rather than one at a time, and while the database is down every Grafana instance reports unhealthy and the endpoint returns 503. The first upgrade after any install re-applies every tier once, even when nothing changed; later upgrades that only touch the Grafana workloads roll without downtime.
An upgrade that adds a new secret reference (enabling smtp with a password, adding datasources.credentialSecrets, turning on backups) can pause the rollout for several minutes while Helm reports success, with The identity ... is not allowed to reveal the secret ... in the deployment status even though the grant is in place. It clears on its own; do not edit policies. Confirm the new version is live with cpln workload get-deployments RELEASE_NAME-grafana --gvc GVC_NAME -o yaml rather than trusting the Helm output.
Rotating a prerequisite secret does not restart anything. After changing a secret’s value, run cpln workload force-redeployment RELEASE_NAME-grafana --gvc GVC_NAME (and the same for RELEASE_NAME-grafana-alerting in dedicated-evaluator mode) so the instances pick it up. Never rotate the encryption key.
Removing a location means removing it from global.locations and from the GVC. Never suspend a location on these workloads; use the values list.
Uninstall deletes every volume set the chart created — the database, the consensus store and, with alerting HA on, Redis and Sentinel. The GVC is yours and is left alone; the prerequisite secrets you created survive. Enable backups if the data matters.

Troubleshooting

A workload sits at 0 replicas after install while helm reported success

Symptom: cpln helm install succeeded, but RELEASE_NAME-grafana (and possibly the database or Redis tiers) show no containers and cpln logs returns nothing.Cause: A prerequisite secret named in your values does not exist — the admin password, the encryption key, the database credentials, or a redisML.*.passwordSecretName. The deployment pauses on the unresolved reference.Fix: Read the message naming the secret, then create it:

cpln workload get-deployments RELEASE_NAME-grafana --gvc GVC_NAME -o yaml

The message reads

The secret <name> no longer exists. Workload updates are paused until the secret is added or the reference to the secret removed.

Once the secret exists the deployment recovers on its own, or run cpln workload force-redeployment RELEASE_NAME-grafana --gvc GVC_NAME to skip the wait.

WARNING locations declared in global.locations are not in GVC

Symptom: The Grafana logs carry [grafana] WARNING: locations declared in global.locations are not in GVC '...', and a location you expected to serve traffic runs nothing.Cause: global.locations names a location the GVC does not have. The platform accepts and stores it without error, so nothing runs there and no deployment reports a failure. On a fresh install the database tier refuses to bootstrap in that state and crash-loops with a FATAL: line naming the location; on an install whose GVC lost a location later, the stack keeps serving and only this warning shows.Fix: Add the location to the GVC or remove it from global.locations, then upgrade. If the missing location is alerting.location, the RELEASE_NAME-grafana-alerting workload holds zero replicas everywhere and no alert rule is evaluated while every dashboard looks healthy — the warning says so explicitly. Check with:

cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-grafana"} |= "[grafana]"' --limit 50 --since 10m

Alert rules never fire but dashboards are healthy

Symptom: Rules exist and show no evaluation timestamp; no contact point ever receives anything.Cause: Either alerting.enabled is false, or you are in dedicated-evaluator mode and alerting.location is a location the GVC does not have (see the previous entry), or that location has been lost.Fix: Set alerting.enabled: true; repoint alerting.location to a surviving location listed in both your values and the GVC and upgrade; or enable alerting.highAvailability.enabled so evaluation runs in every location.

Every alert is delivered twice with alerting HA on

Symptom: Each firing alert produces one notification per Grafana instance.Cause: The coordination tier is unhealthy or unreachable: the RELEASE_NAME-redis or RELEASE_NAME-sentinel workloads are down, fewer than a majority of the configured locations exist in the GVC, or a workload-list firewall on the Redis tier does not include the Grafana workloads. Grafana logs Failed to look up position, falling back to position 0. It never goes silent, so the alert rules themselves are fine.Fix: Check those workloads with cpln workload get-deployments RELEASE_NAME-redis --gvc GVC_NAME -o yaml, restore the missing locations, or add the printed links to redisML.firewall.workloads. A brief duplicate window right after a helm upgrade is expected and clears on its own.

Failed to lock database on a cold start

Symptom: During a fresh install or a tier restart, one or more Grafana instances exit once with Failed to lock database and are restarted.Cause: Grafana runs its schema migrations under a non-blocking lock; an instance arriving while another holds it exits, and the next attempt succeeds against the migrated schema. The chart staggers the instances so this is rare at one replica per location; two replicas in the same location can still race.Fix: None needed. If it keeps happening for minutes, check that the database primary landed in postgresML.primaryLocation — migrations run across regions otherwise and take longer.

failed to send email unencrypted connection

Symptom: No alert emails arrive; the sending workload logs failed to send email: unencrypted connection.Cause: smtp.user is set and the relay does not offer STARTTLS or TLS. Grafana refuses to send credentials over an unencrypted connection and drops the notification.Fix: Use a relay that offers STARTTLS or TLS, or leave smtp.user empty to send unauthenticated against a plain relay.

A provisioned datasource reports upstream connect error connection timeout

Symptom: Panels on a datasource from datasources.definitions show only upstream connect error ... connection timeout.Cause: Almost always an unsubstituted placeholder in the datasource URL — the workload name or the GVC segment of http://YOUR_WORKLOAD.YOUR_GVC.cpln.local:PORT was left literal. A name that does not resolve times out rather than failing fast, so it looks like a network problem.Fix: Correct datasources.definitions[].url and upgrade.

Dashboards load but every panel is empty

Symptom: On a per-location hostname the UI and dashboard layout load, but no panel shows data and no error appears.Cause: Grafana’s root URL is the canonical endpoint, and panel data requests from another hostname are rejected.Fix: Use the canonical *.cpln.app endpoint from status.canonicalEndpoint. It is georouted and already serves from the nearest location.

Render fails with the global.gvc values key was REMOVED

Symptom: helm install or helm upgrade aborts before touching anything, with a message naming etcd-multi-location, postgres-multi-location or this chart and the text the global.gvc values key was REMOVED.Cause: Your values file is from 1.x, which named a GVC under global.gvc.Fix: Do not retry as an upgrade. Follow Migrating From Version 1.

Render fails because the database connection budget is exceeded

Symptom: The install refuses with the database connection budget is exceeded — N Grafana instances ... above the 80-connection ceiling.Cause: (replicas × locations + 1) × database.maxOpenConn is above 80.Fix: Lower database.maxOpenConn or lower replicas; the message spells out the arithmetic.

Render fails because a subchart workload-list does not include the Grafana workloads

Symptom: The install refuses with postgresML.internalAccess.type is 'workload-list' but ... does not include or the redisML.firewall.internalAllowType equivalent.Cause: The database and Redis tiers keep their own firewall lists and do not add Grafana automatically; without the links Grafana could never reach its database or its alerting coordinator.Fix: Paste the links printed in the message into postgresML.internalAccess.workloads or redisML.firewall.workloads.

Important Notes

Create the admin password, encryption key and database credentials secrets before installing. Without them the deployment waits silently on secrets that do not exist while helm install reports success.
There is no upgrade path from 1.x. A helm upgrade across the 2.0.0 boundary deletes the old GVC and everything in it. Follow Migrating From Version 1.
Every location in global.locations must already exist in the GVC you install into, and alerting.location and postgresML.primaryLocation must be among them. GVC locations you do not list run nothing.
Never rotate or delete the encryption key. Every stored datasource credential becomes unreadable in every location, and alert rules that query them fail.
Treat every helm upgrade that touches the database tier as a planned outage, and expect the first upgrade after any install to re-apply every tier.
Alert evaluation stops if you lose alerting.location, and the UI will not show it. Repoint the knob and upgrade, or enable alerting HA — after weighing its query-load cost.
Alerting HA multiplies data-source query load by (locations × replicas) and is blocked below 3 locations on purpose.
With alerting HA on, an unhealthy Redis tier means duplicate notifications, never silence — check the RELEASE_NAME-redis and RELEASE_NAME-sentinel workloads before suspecting your rules.
The alerting-HA Redis tier is unauthenticated by default. Authenticate it unless you control everything in the GVC — see Authenticating the Redis Tier.
Setting a subchart firewall to workload-list requires listing the Grafana workloads in it; the chart refuses to render otherwise and prints the links.
Use the canonical *.cpln.app endpoint, not a per-location hostname — panels stay empty otherwise.
Set postgresML.primaryLocation where most of your users are. Grafana has no read/write splitting, so every other location pays a cross-region round trip per query.
After rotating a prerequisite secret, force a redeployment of the Grafana workloads; nothing picks the new value up on its own.
Uninstall deletes the database volume sets. Enable backups if the data matters; your prerequisite secrets survive.
This template ships Grafana OSS only — Enterprise features such as fine-grained RBAC, reporting and query caching are not available.

External References

Grafana Documentation

Official Grafana documentation

Grafana Alerting

Alert rules, contact points, notification policies, and silences

Alerting High Availability

How Grafana coordinates alert evaluation across instances

Provisioning Datasources

Reference for the datasource provisioning entries used by this template

Configuration Reference

Every Grafana setting and its environment-variable name

Template Catalog

Templates

Overview

What Gets Created

Prerequisites

Installation

UI

CLI

Terraform

Pulumi

Configuration

Locations

Grafana UI Tier

Alerting

Admin Credentials and Encryption Key

Datasources as Code

SMTP

Access

App Database

Alerting HA Coordination

Authenticating the Redis Tier

Connecting

First Run

Operations

Backing Up

Restoring a Backup

Migrating From Version 1

Alert Evaluation

Scaling and Availability

Troubleshooting

Important Notes

External References

Grafana Documentation

Grafana Alerting

Alerting High Availability

Provisioning Datasources

Configuration Reference

​Overview

​What Gets Created

​Prerequisites

​Installation

UI

CLI

Terraform

Pulumi

​Configuration

​Locations

​Grafana UI Tier

​Alerting

​Admin Credentials and Encryption Key

​Datasources as Code

​SMTP

​Access

​App Database

​Alerting HA Coordination

​Authenticating the Redis Tier

​Connecting

​First Run

​Operations

​Backing Up

​Restoring a Backup

​Migrating From Version 1

​Alert Evaluation

​Scaling and Availability

​Troubleshooting

​Important Notes

​External References

Grafana Documentation

Grafana Alerting

Alerting High Availability

Provisioning Datasources

Configuration Reference

Overview

What Gets Created

Prerequisites

Installation

Configuration

Locations

Grafana UI Tier

Alerting

Admin Credentials and Encryption Key

Datasources as Code

SMTP

Access

App Database

Alerting HA Coordination

Authenticating the Redis Tier

Connecting

First Run

Operations

Backing Up

Restoring a Backup

Migrating From Version 1

Alert Evaluation

Scaling and Availability

Troubleshooting

Important Notes

External References