Skip to main content

Overview

Grafana Multi-Location deploys one logical Grafana whose UI/API instances run in every location of a chart-created multi-location GVC, behind a single georouted *.cpln.app endpoint. Every dashboard, user, org, session, alert rule and saved datasource lives in a postgres-multi-location cluster stretched across the same locations, so Grafana itself holds no state: there is no volume, no session affinity and nothing to hand over when an instance is replaced. Alert rules are evaluated by a separate single-replica workload that runs in exactly one location with rule execution enabled, while it is disabled on every UI instance. Exactly-once evaluation is therefore a property of the topology — nothing is elected at runtime, and no value of replicas can produce a second evaluator. See Alert Evaluation.
For a Grafana inside a single location, use the Grafana template instead — it also offers multi-replica alerting HA coordinated by Redis Sentinel, which this template does not.

How Many Locations You Need

The bundled database’s consensus store (etcd-multi-location) commits a write only when a majority of its members agree, and it runs one member per location. That arithmetic decides what survives — the Grafana tier itself has no quorum of its own. With N locations you survive floor((N-1)/2) losses, so an even count buys nothing over the odd count below it. Losing the location named in alerting.location is a separate matter — see Alert Evaluation.

What Gets Created

  • GVC — A new GVC pinned to the configured locations. This template always creates it.
  • Standard Grafana UI Workload{release}-grafana-ml, replicas instances per location, serving the UI and HTTP API on port 3000. Public by default. Alert rule execution is disabled here.
  • Standard Alert Evaluator Workload{release}-grafana-ml-alerting, the same image with exactly one replica, running only in alerting.location and never reachable from the internet. Optional — set alerting.location: "" to omit it entirely.
  • App Database Workloads — The postgres-multi-location subchart: a stateful Patroni PostgreSQL workload with one primary and asynchronous replicas, a stateful etcd workload for consensus, and an HAProxy leader-routing tier in every location.
  • Volume Sets{release}-postgres-ml-vs for the PostgreSQL data directory and the etcd cluster’s own volume set. Grafana has none.
  • Secrets — The subchart’s startup scripts, and — only when datasources.definitions is set — {release}-grafana-ml-datasources, the rendered datasource provisioning file mounted by both Grafana workloads. The admin password, encryption key and database credentials are prerequisite secrets you create yourself; the chart references them by name and never creates, modifies or deletes them.
  • Identity & Policy — One identity shared by both Grafana workloads, with reveal on exactly the secrets they mount and nothing else.
This template creates its own GVC. You do not need to create one before installing.

Prerequisites

The GVC named in global.gvc.name must not already exist. Helm adopts a GVC that does, and helm uninstall then deletes it along with every unrelated workload in it. Always point this template at a fresh GVC name.
Three secrets must exist before you install. The admin account is a human-facing login and publicAccess.enabled defaults to true, so its password never passes through Helm values; the encryption key and the database credentials are shared by every instance in every location.
1

Create the first-boot admin password

An opaque secret with encoding plain:
Set admin.passwordSecretName to the name you used.
2

Create the datasource encryption key

An opaque secret with encoding plain. Every instance reads it on every boot to decrypt datasource credentials stored in the shared database — back it up outside Control Plane and never rotate it:
Set admin.secretKeySecretName to the name you used.
3

Create the app database credentials

A dictionary secret holding exactly username, password and database:
Use plain identifiers for username and database — they are used unquoted when the database is created. Set postgresML.postgres.credentialsSecretName to the secret’s name.
4

Read a secret back later

Without -o json the command prints a table containing no secret data.
A missing prerequisite secret wedges the install rather than failing it. cpln helm install still reports success while the affected workload sits at zero replicas waiting on a secret reference that never resolves, which looks like a broken install. Create all three first, and confirm with cpln workload get-deployments {release}-grafana-ml --gvc {gvc} rather than trusting the Helm output.
Optional features each need something created before you install: Once the secrets exist, install the template using your preferred method:

UI

Browse, install, and manage templates visually

CLI

Manage templates from your terminal

Terraform

Declare templates in your Terraform configurations

Pulumi

Declare templates in your Pulumi programs

Configuration

The default values.yaml for this template:

GVC and Locations

The GVC name and location list live under global.gvc so that Helm passes the same values to the bundled database and, through it, to etcd. Never configure the location lists separately.
  • global.gvc.name — Name of the GVC this chart creates. It must not already exist.
  • global.gvc.locations[].name — A Control Plane location (e.g. aws-us-east-1). At least 2 are required, and the chart refuses to render with fewer.
  • global.gvc.locations[].replicasDatabase members in that location, not Grafana instances. It must be at least 1; removing a location from the list is the supported way to shrink the deployment.

Grafana UI Tier

  • image — The Grafana OSS container image. An Alpine-based variant is required: the boot wrapper needs a shell.
  • replicas — Grafana UI instances per location. It carries no alerting-related restriction, because alert evaluation is a separate workload — the only thing to watch is the connection budget below.
  • resourcesminCpu/minMemory are the reservation, maxCpu/maxMemory the limit, applied per instance.
  • database.maxOpenConn — Maximum database connections per instance.
(replicas × locations + 1) × database.maxOpenConn must stay at 80 or less — the bundled cluster’s max_connections is 100, and the remainder is headroom for Patroni and administration. The + 1 is the alert evaluator. The chart enforces the budget at render time and refuses to install with the arithmetic spelled out, so raising replicas past that point means lowering database.maxOpenConn (or lowering replicas).
Signup, anonymous access and upstream analytics are disabled by the template and are not configurable.

Admin Credentials and Encryption Key

Both credentials live in opaque secrets you create before installing (see Prerequisites) — neither ever passes through Helm values. They sit next to each other in values.yaml but their lifecycles are opposites: Both workloads carry the admin bootstrap environment deliberately. Grafana’s built-in default password is the literal string admin, so an evaluator without it that won the race for the empty database would create an admin/admin account on a publicly exposed UI. Testing confirms that does not happen: admin/admin is refused with a 401 on a fresh install.
The encryption key is permanent. Every instance in every location decrypts saved datasource credentials with it, so changing its payload makes all of them unreadable everywhere — and alert rules that query those datasources then fail. Back the key up outside Control Plane instead of rotating it.

Datasources as Code

datasources.definitions entries are standard Grafana datasource provisioning entries, rendered verbatim into a provisioning file that every instance of both workloads mounts at boot. Applying the same file concurrently from every instance is safe — testing at ten instances produced exactly one row per definition. Credentials never go into the provisioning file. Put them in a dictionary secret created before installing, list it under datasources.credentialSecrets, and reference each key as $KEY:
Each listed key is exposed to both workloads as an environment variable and interpolated when Grafana loads the file; the chart grants the identity reveal on exactly that secret. Provisioned datasources are read-only in the UI — change the value and upgrade to change one.

SMTP

Authenticated SMTP requires a relay that offers STARTTLS or TLS. Grafana refuses to send credentials over an unencrypted connection — it fails with unencrypted connection and every notification is lost, with the only signal a log line in the alerting workload, which is not publicly reachable. Hosted relays (SES, SendGrid, Mailgun, Microsoft 365, Gmail) are unaffected; a plain in-GVC relay is not. Leave smtp.user empty to send unauthenticated against such a relay.
Grafana sends alert notification emails through the SMTP server you configure, and the alert evaluator is what actually sends them. smtp.passwordSecretName is required whenever smtp.user is set, and the password stays in a pre-created opaque secret with encoding plain. Leave smtp.enabled: false if you use webhook or chat contact points instead.

Access

publicAccess.enabled applies to the UI tier only. When it is on, the tier is served on the canonical *.cpln.app HTTPS endpoint and Grafana’s root_url is derived from it automatically, including for the evaluator, so links in delivered notifications point at the UI tier and open in a browser. The evaluator itself is never reachable from the internet — it gets a canonical endpoint but requests to it are refused with a 403.
A firewall change is not instant. Turning public access off was measured at about 107 seconds after the new workload version was serving. Allow up to about four minutes and re-test before concluding a setting did not apply.

App Database

The postgresML block configures the bundled postgres-multi-location cluster: one primary, asynchronous replicas in the other locations, an HAProxy leader-routing tier in each, and automatic promotion in a surviving location. Everything the database tier can do — pooling, restores, emergency quorum recovery, per-member addressing — is documented on that template’s page. postgresML.primaryLocation is a preferred location for the primary: the members elsewhere wait up to 90 seconds for it to initialize the cluster before bootstrapping themselves, and it also biases later elections. Keep it aligned with alerting.location so the evaluator’s queries have no cross-region hop.
Grafana has no read/write splitting — every query, including every dashboard load, goes to the single primary. Testing confirms the shape directly: with seven Grafana instances running, all application connections were on the primary and both standbys carried none. Every location except the primary’s therefore pays one cross-region round trip per query, so set primaryLocation where most of your users are.

Alert Evaluation

Grafana’s memberlist alerting HA coordinates instances over a UDP gossip channel, and UDP is not available between workloads on Control Plane — so if every instance evaluated rules you would get one notification per instance. Grafana also supports a Redis-backed alerting HA path, which would work here; it is deferred to a later version because it needs a stretched Redis tier, not because alerting HA is impossible on the platform. This template makes exactly-once evaluation a property of the topology instead: rule execution is disabled on the UI tier and enabled on a separate workload pinned to one replica in alerting.location, with zero replicas in every other location. Nothing is elected at runtime. Verified in testing at replicas: 3 across three locations: all 9 UI instances report rule execution disabled, the single evaluator reports it enabled, and the other two locations run no evaluator replica at all. Behaviorally, an always-firing rule produced 8 notifications with 8 distinct request IDs over the observation window, every one from the evaluator’s location, with never two in the same instant. Four consequences worth knowing before you rely on it:
  • Losing the evaluator’s replica self-heals. The platform reschedules it and evaluation resumes with no operator action — a measured gap of 21 seconds between the last notification from the old replica and the first from its replacement. The UI tier was unaffected throughout (every probe returned 200).
  • Losing that whole location does not self-heal. Alert evaluation stops until you run helm upgrade with alerting.location set to a surviving location, and the UI gives no sign of it — dashboards look perfectly healthy while nothing is being evaluated.
  • Moving alerting.location does not open a two-evaluator window. Only the evaluator workload is updated; the UI tier is untouched and does not restart. In the measured relocation the old evaluator’s last notification preceded the new evaluator’s readiness by 57 seconds, so the two never overlapped. Expect a gap of a few minutes with no evaluation while the new one boots.
  • Silences do not propagate between instances. Without gossip, a silence created against the UI tier is not guaranteed to be honored by the evaluator, so create silences against the evaluator directly:
The evaluator only answers from inside alerting.location. Its internal name resolves to the GVC address from every location, but the other locations have no local upstream for it and return 503. Run the silence command from a workload replica that is running in alerting.location.
Setting alerting.location: "" renders no evaluator at all — rules can still be created and viewed, and are never evaluated.

Cross-Location Behavior

Every instance shares one database, so there is no replication step between a write in one location and a read in another. Verified across three locations at replicas: 2 — six UI instances plus the evaluator, each result attributed to a named individual instance: Cross-region visibility was bounded at roughly 1–3 seconds by the measurement resolution rather than by the system.
The public endpoint is proximity-routed, not round-robin. In a 100-request tabulation, all 100 requests from one client were served by a single location — the one nearest the client. Several locations give you regional redundancy and locally served traffic, not a request spread across regions.

Connecting

Substitute your release name and the GVC name from global.gvc.name. Health and readiness are served at /api/health, which reports the Grafana version and the app-database status. The Grafana HTTP API is available on the same endpoint for scripted dashboard, datasource and alert-rule management.
An in-GVC request to RELEASE_NAME-grafana-ml.GVC_NAME.cpln.local:3000 is always served by an instance in the caller’s own location. That is why a single write-then-read from one client proves nothing about cross-location state.

Availability and Planned Outages

Measured on three locations (aws-us-east-1, aws-eu-central-1, aws-us-west-2).
Treat every helm upgrade as a planned outage rather than a rolling one. The bundled database members do not restart one at a time — the field that would serialize the rollout is not retained by the platform, so they go down together. Grafana’s readiness probe is /api/health, which reports the database, so while the database is down the whole Grafana tier drops out of the load balancer and returns 503. Changes confined to the Grafana workloads do not have this cost.
An upgrade that adds a new secret reference can pause the rollout for about 9–10 minutes while helm upgrade reports success. Affected locations show The identity ... is not allowed to reveal the secret ... even though the grant is already in place and visible in cpln secret access-report. It clears itself with no action (measured 9 m 0 s – 9 m 30 s); do not re-run the upgrade or start editing policies. Setting admin.applyPassword back to true, enabling smtp with a password, and adding datasources.credentialSecrets are the changes that trigger it. It also means Helm reporting success is not evidence the new version is running — check cpln workload get-deployments.
A single Failed to lock database line during a cold start is expected. Grafana takes a non-blocking advisory lock to run schema migrations and exits if another instance holds it; the platform restarts it and the next attempt succeeds against the already-migrated schema. Seven restarts across the tier were observed on a three-location cold install, all from this race.

Backing Up

Backups are disabled by default and cover the app database — the dashboards, users, alert rules and saved datasources that make up your Grafana. Enable them with postgresML.backup.enabled: true, choose logical (a scheduled pg_dumpall cron workload in the one location named by postgresML.backup.location) or wal-g (continuous archiving from whichever member is currently the primary), and complete the storage setup for your provider before installing.
The backup path has been exercised against AWS S3 and MinIO, and a wal-g restore has been verified end to end (base backup plus WAL replay into an empty data directory, checksum-identical to source). The logical restore and the volume-set swap procedure have not been exercised. The gcp and minio providers are configured the same way but have not been exercised — validate them in a test install, and rehearse your restore procedure, before relying on either.
1

Create a bucket

Create an S3 bucket. Set postgresML.backup.aws.bucket and postgresML.backup.aws.region to match.
2

Set up a Cloud Account

If you do not have one, create a Cloud Account for the AWS account holding the bucket. Set postgresML.backup.aws.cloudAccountName to its name.
3

Create a bucket-scoped IAM policy

Create an IAM policy with the JSON below (replace YOUR_BUCKET_NAME), then set postgresML.backup.aws.policyName to the policy’s name. This bucket-scoped policy is all the identity needs — no broad managed policy is required.
4

Choose a prefix

Set postgresML.backup.aws.prefix to the folder path within the bucket.

Important Notes

  • Create the admin password, encryption key and database credentials secrets before installing. The chart creates none of them; without them the deployment waits on secrets that do not exist while helm install reports success.
  • The GVC in global.gvc.name must not already exist. Helm adopts an existing one and deletes it on uninstall, taking every unrelated workload with it.
  • Never rotate or delete the encryption key. Every instance in every location decrypts stored datasource credentials with it; changing it makes them all unreadable and alert rules that query them fail.
  • Every helm upgrade that touches the database tier is a planned outage. Database writes stop for about 117 s and the Grafana tiers return errors for longer — measured recovery about 4 minutes, one location 5–6 minutes. The first upgrade after any install costs this even for a Grafana-only change. Changes confined to the Grafana workloads roll with zero downtime.
  • An upgrade that adds a secret reference can pause the rollout for about 9–10 minutes while Helm reports success. It clears itself — do not re-run the upgrade or edit policies.
  • Alert evaluation stops if you lose alerting.location, and the UI will not show it. Repoint the knob and upgrade; that restarts only the evaluator.
  • Silences must be created against the evaluator, from a workload in alerting.location — its internal address returns 503 from every other location.
  • The public endpoint is proximity-routed. More locations means regional redundancy and locally served traffic, not requests spread across regions.
  • Every location except the database primary’s pays a cross-region round trip per query, because Grafana has no read/write splitting. Set postgresML.primaryLocation where most of your users are.
  • Scaling replicas has no alerting-related restriction — it applies to the UI tier only, in every location including alerting.location. Watch the connection budget instead.
  • replicas: 2 or higher lengthens a cold install to roughly 7 minutes per location and up to about 12 minutes for every replica. Nothing is wrong; there is simply more to schedule.
  • Grafana Live has no HA engine here, so a live-streamed message reaches only the browsers connected to the same instance. Dashboard auto-refresh, queries, alerting, provisioning, login and the API are unaffected.
  • With publicAccess.enabled: false, links in alert notifications point at the internal GVC address and will not open from a browser outside the GVC.
  • Never suspend a location. Suspending and resuming one permanently withdraws its endpoints from the other locations’ service discovery while every status surface still reports healthy. Remove a location from global.gvc.locations instead.
  • Uninstall deletes the database volume sets along with the GVC — enable backups if the data matters. Prerequisite secrets you created are not owned by the release and survive it.
  • This template ships Grafana OSS only — Enterprise features such as fine-grained RBAC, reporting and query caching are not available.

External References

Grafana Documentation

Official Grafana documentation

Grafana Alerting

Alert rules, contact points, notification policies, and silences

Alerting High Availability

How Grafana coordinates alert evaluation across instances

Provisioning Datasources

Reference for the datasource provisioning entries used by this template

Configuration Reference

Every Grafana setting and its environment-variable name

Grafana HTTP API

Manage dashboards, datasources, and alert rules programmatically

Grafana Multi-Location Template

View the source files, default values, and chart definition