Grafana Multi-Location

​

Overview

• Off (the default) — a separate single-replica workload runs in one location with rule execution enabled, and it is disabled on every UI instance. Exactly-once evaluation is a property of the topology: nothing is elected at runtime, and no value of replicas can produce a second evaluator. Losing that location stops evaluation.
• On — that workload is not created at all. Every UI instance in every location evaluates every rule, and a stretched Redis tier coordinates the peers so exactly one notification is sent. Alert evaluation then survives the loss of a location, at the cost of multiplying data-source query load.

​

How Many Locations You Need

​

What Gets Created

• Standard Grafana UI Workload — {release}-grafana, replicas instances per location, serving the UI and HTTP API on port 3000. Public by default. Alert rule execution is disabled here by default, and enabled on every instance when alerting HA is on.
• Standard Alert Evaluator Workload (optional) — {release}-grafana-alerting, the same image with exactly one replica, running only in alerting.location and never reachable from the internet. Not created when alerting.highAvailability.enabled is true, or when alerting.enabled is false.
• App Database Workloads — The postgres-multi-location subchart: a stateful Patroni PostgreSQL workload with one primary and asynchronous replicas, a stateful etcd workload for consensus, and an HAProxy leader-routing tier in every location.
• Alerting Coordination Workloads (optional) — The redis-multi-location subchart: {release}-redis and {release}-sentinel, one of each per location, coordinating exactly-once notification delivery. Created only when alerting.highAvailability.enabled is true.
• Volume Sets — {release}-postgres-vs for the PostgreSQL data directory and the etcd cluster’s own volume set, plus {release}-redis-vs and {release}-sentinel-vs with alerting HA on. Grafana itself has none.
• Secrets — The subchart startup scripts and Redis configuration, and — only when datasources.definitions is set — {release}-grafana-datasources, the rendered datasource provisioning file mounted by every Grafana workload. The admin password, encryption key and database credentials are prerequisite secrets you create yourself; the chart references them by name and never creates, modifies or deletes them.
• Identities & Policies — One identity shared by every Grafana workload, with reveal on exactly the secrets they mount and nothing else, plus a second policy granting view on exactly the one GVC this release installs into so the boot check can read that GVC’s location list. Each subchart tier gets its own identity and policy.

​

Prerequisites

printf '%s' "$(openssl rand -hex 24)" \
  | cpln secret create-opaque --name my-grafana-admin-password --encoding plain -f -

printf '%s' "$(openssl rand -hex 32)" \
  | cpln secret create-opaque --name my-grafana-secret-key --encoding plain -f -

cpln secret create-dictionary --name my-grafana-db-credentials \
  --entry username=grafana \
  --entry password="$(openssl rand -hex 24)" \
  --entry database=grafana

cpln secret reveal my-grafana-db-credentials -o json

​

Configuration

# ─── Locations ────────────────────────────────────────────────────────────────
# This chart deploys into the GVC you install into — it does NOT create one.
# Every location listed here MUST already exist in that GVC. The platform does
# not validate that: a location the GVC lacks is stored verbatim and is simply
# inert, so nothing runs there and no deployment reports a failure. Both Grafana
# tiers read the GVC at boot and log a warning naming the mismatch. Extra
# locations in the GVC are fine — nothing here runs in them.
#
# Lives under `global` so BOTH subcharts get the same list automatically — the
# postgres-multi-location database (and its own etcd, two levels down) and the
# optional redis-multi-location alerting coordinator. Never maintain two lists.
#
# Minimum 2 locations, and minimum 3 with alerting HA enabled below. The
# database tier needs 3 for automatic failover and 5 to survive losing two —
# see the survival table in the README.
# `replicas` here sets the POSTGRES members in that location, and nothing else.
# Every other tier ignores it and has its own count:
#   etcd     — always exactly 1 per location, not configurable. Quorum is a
#              majority of LOCATIONS, so a second member in one buys nothing.
#   Grafana  — the top-level `replicas` below, applied to every location.
#   Redis    — redisML.redis.replicasPerLocation (alerting HA only).

global:
  locations:
    - name: aws-us-east-1
      replicas: 1
    - name: aws-eu-central-1
      replicas: 1
    - name: aws-us-west-2
      replicas: 1

# ─── Grafana UI tier ──────────────────────────────────────────────────────────
image: grafana/grafana:13.1.3

replicas: 1 # Grafana UI instances PER LOCATION

resources:
  maxCpu: 1000m
  maxMemory: 1Gi
  minCpu: 500m
  minMemory: 512Mi

database:
  maxOpenConn: 10 # per instance; see the README budget

# ─── Alerting ─────────────────────────────────────────────────────────────────
# enabled: false turns rule evaluation off entirely — rules can still be created
# and viewed, nothing ever evaluates them.
#
# highAvailability.enabled: false (default) — a SEPARATE single-replica workload
# in `location` evaluates every rule, and exactly-once follows from there being
# only one of it. Restarting it re-notifies whatever is firing at the time, and
# losing that location stops evaluation with no sign of it in the UI.
#
# highAvailability.enabled: true — no separate workload; every UI instance
# evaluates and a stretched Redis (redisML below) picks exactly one to SEND, so
# alerting survives losing a location. Needs 3+ locations. The costs: datasource
# query load multiplies by (locations × replicas) because Redis dedupes the
# notification and not the query; a large rule set may need resources.maxCpu
# raised; and if Redis is unhealthy alerting DUPLICATES rather than going silent.
alerting:
  enabled: true
  highAvailability:
    enabled: false
  # Dedicated-evaluator mode ONLY — where the one evaluator runs. Must be one of
  # global.gvc.locations. IGNORED when highAvailability.enabled is true.
  location: aws-us-east-1
  # Dedicated-evaluator mode ONLY — in HA mode the UI tier's `resources` apply.
  resources:
    maxCpu: 1000m
    maxMemory: 1Gi
    minCpu: 500m
    minMemory: 512Mi

# ─── Admin Bootstrap & Encryption (prerequisite secrets) ──────────────────────
# BOTH secrets must EXIST BEFORE INSTALL — opaque, encoding: plain (see README).
# The admin login is on the public internet when publicAccess.enabled is true,
# so its password never transits values or the Helm release.
admin:
  user: admin # initial admin login name (not sensitive)
  # The password applies only when the admin account is FIRST created; on later
  # boots Grafana ignores it (change it in the UI instead).
  applyPassword: true # set false after your first login to stop referencing the password secret
  passwordSecretName: my-grafana-admin-password # opaque secret holding the first-boot admin password
  # DIFFERENT LIFECYCLE — permanent. Encrypts datasource passwords AT REST in the
  # shared database; it protects nothing in transit (that is TLS, via the
  # datasource URL). Every instance in every location must use the SAME key, or a
  # datasource saved in one region is undecryptable in another and rules querying
  # it fail silently. Rotating it invalidates every credential already stored, so
  # never delete it and never rotate it.
  secretKeySecretName: my-grafana-secret-key # opaque secret holding the at-rest encryption key

# ─── Datasources as Code (optional) ───────────────────────────────────────────
# Grafana datasource provisioning entries, passed through verbatim. Every
# instance of every workload applies the same file on boot.
datasources:
  definitions: []
  # - name: Prometheus
  #   type: prometheus
  #   access: proxy
  #   # Substitute BOTH parts: the workload name AND the GVC. A leftover
  #   # placeholder resolves to nothing and the panel shows only
  #   # "upstream connect error ... connection timeout".
  #   url: http://YOUR_WORKLOAD.YOUR_GVC.cpln.local:9095
  #   isDefault: true
  # - name: AppDB
  #   type: postgres
  #   url: my-db-host:5432
  #   user: grafana_reader
  #   jsonData: { database: appdb, sslmode: disable }
  #   secureJsonData:
  #     password: $PG_PASSWORD # interpolated from credentialSecrets below

  # The credentials Grafana authenticates TO each datasource with. `definitions`
  # renders into a plaintext provisioning file, so put the password in a
  # pre-created dictionary secret and write $KEY above. Every $KEY needs an entry
  # here, and the secrets MUST EXIST BEFORE INSTALL.
  credentialSecrets: []
  # - name: my-grafana-ds-credentials
  #   keys: [PG_PASSWORD]

# ─── SMTP for Alert Emails (optional) ─────────────────────────────────────────
# Whichever instance evaluates is what actually sends: the dedicated evaluator
# workload by default, or the coordinating UI instance when alerting HA is on.
smtp:
  enabled: false
  host: smtp.example.com:587 # host:port
  user: "" # empty = unauthenticated SMTP. IF YOU SET THIS, the relay MUST offer
  # STARTTLS or TLS: Grafana refuses to send credentials over an unencrypted
  # connection ("failed to send email: unencrypted connection") and every
  # notification is then lost, with the only signal a log line in the
  # sending workload. Hosted relays (SES, SendGrid, Mailgun, M365, Gmail)
  # are fine; a plain in-GVC relay is not — leave this empty for those.
  passwordSecretName: "" # opaque secret (encoding: plain) with the SMTP password; create BEFORE install
  fromAddress: grafana@example.com
  fromName: Grafana

# ─── Access ───────────────────────────────────────────────────────────────────
# Applies to the UI tier. The dedicated evaluator workload is NEVER public; it
# honours internalAccess only, so you can reach it in-GVC to manage silences.
publicAccess:
  enabled: true # UI on the canonical *.cpln.app HTTPS endpoint

internalAccess:
  type: same-gvc # options: none, same-gvc, same-org, workload-list
  # Only used when type is workload-list. This chart's OWN workloads (the UI
  # tier and, when it renders, the alert evaluator) are added automatically —
  # the list also governs the in-GVC path the README's silence procedure uses,
  # so a list naming only clients would deny it. List your clients here; do not
  # list this release's workloads.
  workloads: []
  # workloads:
  #   - //gvc/GVC_NAME/workload/WORKLOAD_NAME

# ─── App Database (subchart: postgres-multi-location) ─────────────────────────
# One Patroni cluster stretched across the same locations: a single primary,
# async replicas elsewhere, automatic promotion in a surviving location.
# Grafana has no read/write splitting, so EVERY query goes to the primary —
# put it where most of your users are.
postgresML:
  # The stretched etcd cluster behind Patroni. Compaction is on by default and
  # should stay on: without it etcd's backend grows with time alone — Patroni
  # renews its lease every ~10s and every renewal is a revision — until it hits
  # the quota and etcd goes READ-ONLY, taking the database's failover with it.
  etcd:
    tuning:
      autoCompactionMode: periodic # periodic (retention is a duration) or revision (a revision count)
      autoCompactionRetention: 1h # periodic needs an explicit unit (1h, 30m, 24h)
      quotaBackendBytes: 0 # backend size limit in bytes; 0 = etcd's own default of 2 GiB

  postgres:
    # REQUIRED PREREQUISITE SECRET — CREATE IT BEFORE YOU INSTALL.
    # A `dictionary` secret holding exactly `username`, `password` and
    # `database`. If it does not exist the deployment wedges waiting on it.
    credentialsSecretName: my-grafana-db-credentials

  # The database tier's own internal firewall. Leave it at same-gvc unless you
  # have a reason not to: this chart's Grafana workloads are NOT added to the
  # subchart's list automatically (a parent cannot inject a rendered name into a
  # subchart's values), so switching to workload-list without listing them cuts
  # Grafana off from its own database. The chart refuses to render in that state
  # and prints the exact links to add.
  internalAccess:
    type: same-gvc # options: same-gvc, same-org, workload-list
    workloads: []

  # Preferred location for the database primary. Keep it aligned with
  # alerting.location so the hot path has no cross-region hop. With alerting HA
  # on, this also decides which Grafana location runs the schema migrations.
  primaryLocation: aws-us-east-1

  resources:
    minCpu: 500m
    minMemory: 1Gi
    maxCpu: 1
    maxMemory: 2Gi

  volumeset:
    capacity: 10 # initial capacity in GiB per member (minimum is 10)

  proxy:
    minReplicas: 2 # HAProxy leader-routing tier, per location
    maxReplicas: 2

  backup: # optional database backups — see Storage setup in the README
    enabled: false
    mode: logical # logical or wal-g
    location: aws-us-east-1 # logical mode only: the ONE location the nightly job runs in
    resources:
      cpu: 100m
      # 512Mi, matching postgres-multi-location's own default. At 128Mi the GCP
      # path OOMs with NO log output: logical jobs merely report `failed` and the
      # wal-g sidecar loops on OOMKilled while WAL archives with no base backup.
      # Do not lower this without re-testing the GCS path.
      memory: 512Mi
    logical:
      image: ghcr.io/controlplane-com/backup-images/postgres-backup:17.1.0
      schedule: "0 2 * * *"
    walg:
      intervalSeconds: 21600
    provider: aws # options: aws, gcp, minio
    aws:
      bucket: my-grafana-bucket
      region: us-east-1
      cloudAccountName: my-s3-cloud-account
      policyName: my-grafana-backup-policy
      prefix: grafana/backups
    gcp:
      bucket: my-grafana-bucket
      cloudAccountName: my-gcs-cloud-account
      prefix: grafana/backups
    minio:
      endpoint: http://my-minio-workload:9000
      bucket: my-grafana-bucket
      credentialsSecretName: my-grafana-minio-credentials
      prefix: grafana/backups

# ─── Alerting-HA coordination (subchart: redis-multi-location) ────────────────
# Rendered ONLY when alerting.highAvailability.enabled is true. One Redis and one
# Sentinel per location; failover needs a majority of them, which is why alerting
# HA requires 3+ locations.
redisML:
  # Which server the coordination tier runs. `redis` changes nothing. `valkey`
  # runs BOTH the Redis and Sentinel tiers on valkeyImage — the BSD-licensed
  # fork of Redis 7.2, which ships redis-server/redis-cli/redis-sentinel
  # compatibility symlinks, so Grafana's Sentinel client is unchanged. Chosen at
  # INSTALL time: a data directory cannot be moved between engines.
  engine: redis # redis | valkey
  valkeyImage: valkey/valkey:8.1.9 # used for both tiers when engine is valkey

  # Same rule as postgresML.internalAccess above: Grafana is not added to this
  # list automatically, and blocked from Sentinel it boots fine and every
  # instance sends its own copy of every alert. The chart refuses to render if
  # you set workload-list without listing the Grafana workloads.
  firewall:
    internalAllowType: same-gvc # options: same-gvc, same-org, workload-list
    workloads: []

  redis:
    # OPTIONAL hardening, off by default. Opaque secret (encoding: plain) whose
    # payload is the password; it MUST EXIST BEFORE INSTALL, and Grafana reads the
    # same one, so a misspelt name stops the UI tier too. "" leans on the same-GVC
    # firewall instead — reasonable when nothing untrusted shares the GVC, which
    # you now choose rather than the chart choosing for you.
    passwordSecretName: ""
    image: redis:7.4
    replicasPerLocation: 1 # Redis members per location; 1 is plenty for alert coordination
    resources:
      cpu: 200m
      memory: 256Mi
    volumeset:
      initialCapacity: 10 # GiB per member (platform minimum); coordination data is tiny
  sentinel:
    # Same shape and same failure mode as redis.passwordSecretName, independent of
    # it — Sentinel is what Grafana asks for the current master.
    passwordSecretName: ""
    image: redis:7.4
    resources:
      cpu: 200m
      memory: 256Mi

​

Locations

• global.locations[].name — A Control Plane location (e.g. aws-us-east-1) that must already be one of the locations of the GVC you install into. At least 2 are required, at least 3 with alerting.highAvailability.enabled: true, and the chart refuses to render with fewer.
• global.locations[].replicas — Database members in that location, not Grafana instances. It must be at least 1; removing a location from the list is the supported way to shrink the deployment. Grafana’s own count per location is the top-level replicas, etcd always runs exactly one member per location, and the Redis tier has its own count (redisML.redis.replicasPerLocation) — all three ignore this number.

​

Matching the Location List to the GVC

cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-grafana"}' | grep '\[grafana\]'

​

Grafana UI Tier

• image — The Grafana OSS container image. It must provide /bin/bash: the chart overrides the entrypoint to run Grafana’s /run.sh through a boot wrapper. The default Alpine-based official image does.
• replicas — Grafana UI instances per location. It carries no alerting-related restriction in either mode — the only thing to watch is the connection budget below.
• resources — minCpu/minMemory are the reservation, maxCpu/maxMemory the limit, applied per instance.
• database.maxOpenConn — Maximum database connections per instance.

​

Admin Credentials and Encryption Key

​

Datasources as Code

datasources:
  definitions:
    - name: AppDB
      type: postgres
      url: my-db-host:5432
      user: grafana_reader
      jsonData: { database: appdb, sslmode: disable }
      secureJsonData:
        password: $PG_PASSWORD
  credentialSecrets:
    - name: my-grafana-ds-credentials
      keys: [PG_PASSWORD]

​

SMTP

​

Access

​

App Database

​

etcd History Compaction

​

Alerting-HA Coordination

• redisML.engine — redis (the default, which changes nothing) or valkey, which runs both the Redis and Sentinel tiers on redisML.valkeyImage. Valkey is the BSD-licensed fork of Redis 7.2 and ships redis-server / redis-cli / redis-sentinel compatibility symlinks, so Grafana’s Sentinel client is unchanged. It is an install-time choice: a data directory cannot be moved between engines.
• redisML.valkeyImage — The image used for both tiers when engine is valkey, at which point redisML.redis.image and redisML.sentinel.image are inert.
• redisML.firewall.internalAllowType / .workloads — The coordination tier’s own internal firewall. See the warning under Access before setting it to workload-list.
• redisML.redis.replicasPerLocation — Redis members per location. One is plenty: the data is a few kilobytes of coordination keys.
• redisML.redis.volumeset.initialCapacity — GiB per member, at the platform minimum of 10.
• redisML.{redis,sentinel}.image / .resources — Passed through to the subchart. These blocks expose only a limit, so they use bare cpu and memory.
• redisML.{redis,sentinel}.passwordSecretName — Optional authentication, empty by default. See below.

​

Authenticating the Redis Tier

printf '%s' "$(openssl rand -hex 32)" | cpln secret create-opaque --name my-grafana-redis-password --encoding plain -f -
printf '%s' "$(openssl rand -hex 32)" | cpln secret create-opaque --name my-grafana-sentinel-password --encoding plain -f -

redisML:
  redis:
    passwordSecretName: my-grafana-redis-password
  sentinel:
    passwordSecretName: my-grafana-sentinel-password

​

Alert Evaluation

​

Choosing an Alerting Mode

​

Read This Before Enabling HA

​

Dedicated Evaluator Mode (the Default)

• Losing the evaluator’s replica self-heals, and re-notifies. The platform reschedules it and evaluation resumes with no operator action — a gap of 21 seconds between the last notification from the old replica and the first from its replacement (measured on 1.0.0, before the database gate was extended, so expect this to be a floor rather than a ceiling), with the UI tier unaffected throughout. Each replacement boots with an empty notification log and immediately re-sends whatever is currently firing, so a restart costs duplicate notifications rather than missed ones.
• Losing that whole location does not self-heal. Alert evaluation stops until you run helm upgrade with alerting.location set to a surviving location, and the UI gives no sign of it — dashboards look perfectly healthy while nothing is being evaluated. This is the failure alerting HA exists to remove.
• That silent shape only happens to a location removed from the GVC after the cluster was initialized. A fresh install naming a location the GVC lacks fails loudly instead: alerting.location must also be in global.locations, and the bundled etcd and Patroni tiers refuse to bootstrap on a fresh data directory for any location the GVC does not have, so the whole stack crash-loops with a named error rather than coming up half-working. Either way the boot warning names the consequence explicitly — the evaluator workload holding zero replicas everywhere, and no alert rule ever being evaluated.
• Moving alerting.location does not open a two-evaluator window. Only the evaluator workload is updated; the UI tier is untouched and does not restart. In the measured relocation the old evaluator’s last notification preceded the new evaluator’s readiness by 57 seconds (measured on 1.0.0, same caveat), so the two never overlapped. Expect a gap of a few minutes with no evaluation while the new one boots.
• Silences are not expected to propagate between instances (not exercised in our testing). Without gossip, a silence created against the UI tier is not guaranteed to be honored by the evaluator, so create silences against the evaluator directly:
  curl -u admin:PASSWORD http://RELEASE_NAME-grafana-alerting.GVC_NAME.cpln.local:3000/api/alertmanager/grafana/api/v2/silences
  
  The API requires credentials, so the -u flag is not optional.

​

Redis-Coordinated Alerting HA

• Which location sends is not pinnable. The sender is chosen by sorted peer name, and a standard workload’s name carries a ReplicaSet hash that changes on every rollout — the elected sender moved from aws-us-east-1 to aws-us-west-2 between two installs of the same chart. Do not build routing, filtering or egress-IP allowlisting on the assumption that a particular location sends.
• After a location is lost, hand-off is not instant — and this one is derived, not timed. Two things were measured directly: the Redis peer keys carry a 5-minute TTL, so a dead location’s peers keep their positions until it expires, and a location other than the previous sender was observed taking alerting over on its own after a rollout reordered the peers. The loss event itself has never been observed — no platform primitive simulates a true region outage — so “alerting continues from the survivors and only the sending instance changes” follows from those two measurements rather than from a timed failover. On the same basis, expect roughly 30 s of extra delay before the first notification from a survivor, decaying to zero as the dead peer keys expire (derived from the 5-minute TTL and Grafana’s peer-timeout default, not timed).

​

Cross-Location Behavior

​

Connecting

​

Migrating From Version 1

​

Availability and Planned Outages

​

Backing Up

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:ListBucket",
                "s3:GetObjectVersion",
                "s3:DeleteObjectVersion"
            ],
            "Resource": [
                "arn:aws:s3:::YOUR_BUCKET_NAME",
                "arn:aws:s3:::YOUR_BUCKET_NAME/*"
            ]
        }
    ]
}

cpln secret create-dictionary --name my-grafana-minio-credentials \
  --entry accessKey=MINIO_ACCESS_KEY \
  --entry secretKey=MINIO_SECRET_KEY

​

Important Notes

• Create the admin password, encryption key and database credentials secrets before installing. The chart creates none of them; without them the deployment waits on secrets that do not exist while helm install reports success.
• There is no upgrade path from 1.x. Every 1.x release created its own GVC; 2.0.0 deploys into an existing one, and a helm upgrade across that boundary deletes the old GVC and everything in it, database volumes included. The chart refuses to render on the 1.x global.gvc key, but that guard cannot see an upgrade run with no values at all. Follow Migrating From Version 1.
• Every location in global.locations must already exist in the GVC you install into. A GVC location this release does not list runs nothing, which is harmless. A listed location the GVC lacks makes a fresh install crash-loop in the database tier with a named error, and — for a location removed from a GVC after initialization — silently runs nothing there. See Matching the Location List to the GVC.
• Setting either subchart’s firewall to workload-list requires listing the Grafana workloads in it. The chart refuses to render otherwise and prints the exact links to add; this chart’s own list adds its own workloads for you.
• Never rotate or delete the encryption key. Every instance in every location decrypts stored datasource credentials with it; changing it makes them all unreadable and alert rules that query them fail.
• Every helm upgrade that touches the database tier is a planned outage. The database went unavailable for about 2 m 17 s and the Grafana tier took about 5 minutes to finish rolling every replica. A build whose primary bootstrapped outside primaryLocation was worse — about 4 minutes to recover, one location 5–6 minutes — so treat that as the worst case. The first upgrade after any install costs this even for a Grafana-only change. Changes confined to the Grafana workloads roll with zero downtime.
• An upgrade that adds a secret reference can pause the rollout for about 9–10 minutes while Helm reports success. It clears itself with no action, and the grant is already in place — there is nothing to fix in the policy. Enabling smtp, datasources.credentialSecrets or postgresML.backup are examples that trigger it.
• Alert evaluation stops if you lose alerting.location, and the UI will not show it. Repoint the knob and upgrade; that restarts only the evaluator. Set alerting.highAvailability.enabled: true to remove that failure — read the query-load cost in Alert Evaluation first.
• Silences must be created against the evaluator, from a workload in alerting.location — its internal address returns 503 from every other location. With alerting HA on there is no evaluator and this does not apply.
• alerting.location is required while alerting is on and HA is off, and it is ignored with HA on. It is no longer a disable switch: use alerting.enabled: false to stop rule evaluation entirely, which also fails at render if combined with alerting.highAvailability.enabled: true.
• Turning alerting HA on or off changes which workloads exist. Enabling it deletes the {release}-grafana-alerting workload and adds a Redis and a Sentinel workload per location; disabling it does the reverse. It is a normal helm upgrade, but treat it as a planned change, not a toggle to flip during an incident.
• Alerting HA multiplies data-source query load by (locations × replicas) — 3× at the defaults — because every instance evaluates every rule and Redis dedupes the notification, not the query.
• Alerting HA is blocked below 3 locations, on purpose. Sentinel elects a master by a majority of locations, so at 2 locations losing either one leaves no quorum.
• With alerting HA on, an unhealthy Redis means duplicate notifications, never silence. Check the {release}-redis and {release}-sentinel workloads before suspecting your alert rules.
• With alerting HA on, a helm upgrade costs 80–95 s of duplicate notifications, even an upgrade that changes nothing, and post-outage sender hand-off is not instant — the Redis peer keys’ 5-minute TTL was measured, but the hand-off itself is derived from it rather than timed against a real outage.
• Which location sends notifications is not pinnable with alerting HA on. The sender is chosen by sorted peer name and moved between regions across two installs of the same chart, so do not route, filter or egress-allowlist on it.
• The alerting-HA Redis tier is unauthenticated by default, reachable only from inside the GVC you install into — which since 2.0.0 is a GVC you own and may already share with other workloads. Authenticate it unless you control everything in that GVC: write access to it is enough to suppress alert notifications. See Authenticating the Redis Tier.
• If you do set redisML.redis.passwordSecretName or redisML.sentinel.passwordSecretName, a wrong name stops the Grafana UI too, because Grafana reads the same secret — and helm install still reports success. If a tier sits at 0 replicas after install, read status.versions[].message on the workload.
• Use the canonical *.cpln.app endpoint, not a per-location hostname. Grafana is configured with a single absolute root_url (the canonical endpoint). On a per-location hostname the UI and dashboard layout load, but the POST that fetches panel data is rejected — you get a dashboard with empty panels and no error shown. The canonical endpoint is georouted and already serves from the nearest location.
• A provisioned datasource reporting upstream connect error ... connection timeout is almost always an unsubstituted placeholder in its URL, not a network or firewall problem. Check the GVC segment of datasources.definitions[].url first — a name that does not resolve times out rather than failing fast, and nothing in the Grafana UI names the cause.
• The public endpoint is proximity-routed. More locations means regional redundancy and locally served traffic, not requests spread across regions.
• Every location except the database primary’s pays a cross-region round trip per query, because Grafana has no read/write splitting. Set postgresML.primaryLocation where most of your users are.
• Scaling replicas has no alerting-related restriction — it applies to the UI tier only, in every location including alerting.location. Watch the connection budget instead.
• replicas: 2 or higher lengthens a cold install to roughly five and a half minutes. Nothing is wrong; there is simply more to schedule.
• Grafana Live has no HA engine here, so a live-streamed message reaches only the browsers connected to the same instance — carried over from the single-location Grafana template’s multi-replica behaviour, and not exercised in this template’s own testing. Dashboard auto-refresh, queries, alerting, provisioning, login and the API are unaffected.
• With publicAccess.enabled: false, links in alert notifications point at the internal GVC address and will not open from a browser outside the GVC.
• Never suspend a location. Suspending and resuming one permanently withdraws its endpoints from the other locations’ service discovery while every status surface still reports healthy. To remove a location, take it out of global.locations and out of the GVC.
• Uninstall deletes every volume set the chart created — the database, etcd and, with alerting HA on, Redis and Sentinel. The GVC is yours and is left alone. Enable backups if the data matters. Prerequisite secrets you created are not owned by the release and survive it.
• This template ships Grafana OSS only — Enterprise features such as fine-grained RBAC, reporting and query caching are not available.

​

External References