Overview
Grafana Multi-Location deploys one logical Grafana whose UI/API instances run in every location of a chart-created multi-location GVC, behind a single georouted*.cpln.app endpoint. Every dashboard, user, org, session, alert rule and saved datasource lives in a postgres-multi-location cluster stretched across the same locations, so Grafana itself holds no state: there is no volume, no session affinity and nothing to hand over when an instance is replaced.
Alert rules are evaluated by a separate single-replica workload that runs in exactly one location with rule execution enabled, while it is disabled on every UI instance. Exactly-once evaluation is therefore a property of the topology — nothing is elected at runtime, and no value of replicas can produce a second evaluator. See Alert Evaluation.
For a Grafana inside a single location, use the Grafana template instead — it also offers multi-replica alerting HA coordinated by Redis Sentinel, which this template does not.
How Many Locations You Need
The bundled database’s consensus store (etcd-multi-location) commits a write only when a majority of its members agree, and it runs one member per location. That arithmetic decides what survives — the Grafana tier itself has no quorum of its own.
With N locations you survive
floor((N-1)/2) losses, so an even count buys nothing over the odd count below it. Losing the location named in alerting.location is a separate matter — see Alert Evaluation.
What Gets Created
- GVC — A new GVC pinned to the configured locations. This template always creates it.
- Standard Grafana UI Workload —
{release}-grafana-ml,replicasinstances per location, serving the UI and HTTP API on port3000. Public by default. Alert rule execution is disabled here. - Standard Alert Evaluator Workload —
{release}-grafana-ml-alerting, the same image with exactly one replica, running only inalerting.locationand never reachable from the internet. Optional — setalerting.location: ""to omit it entirely. - App Database Workloads — The
postgres-multi-locationsubchart: a stateful Patroni PostgreSQL workload with one primary and asynchronous replicas, a stateful etcd workload for consensus, and an HAProxy leader-routing tier in every location. - Volume Sets —
{release}-postgres-ml-vsfor the PostgreSQL data directory and the etcd cluster’s own volume set. Grafana has none. - Secrets — The subchart’s startup scripts, and — only when
datasources.definitionsis set —{release}-grafana-ml-datasources, the rendered datasource provisioning file mounted by both Grafana workloads. The admin password, encryption key and database credentials are prerequisite secrets you create yourself; the chart references them by name and never creates, modifies or deletes them. - Identity & Policy — One identity shared by both Grafana workloads, with
revealon exactly the secrets they mount and nothing else.
This template creates its own GVC. You do not need to create one before installing.
Prerequisites
Three secrets must exist before you install. The admin account is a human-facing login andpublicAccess.enabled defaults to true, so its password never passes through Helm values; the encryption key and the database credentials are shared by every instance in every location.
1
Create the first-boot admin password
2
Create the datasource encryption key
An opaque secret with encoding Set
plain. Every instance reads it on every boot to decrypt datasource credentials stored in the shared database — back it up outside Control Plane and never rotate it:admin.secretKeySecretName to the name you used.3
Create the app database credentials
A dictionary secret holding exactly Use plain identifiers for
username, password and database:username and database — they are used unquoted when the database is created. Set postgresML.postgres.credentialsSecretName to the secret’s name.4
Read a secret back later
-o json the command prints a table containing no secret data.
Once the secrets exist, install the template using your preferred method:
UI
Browse, install, and manage templates visually
CLI
Manage templates from your terminal
Terraform
Declare templates in your Terraform configurations
Pulumi
Declare templates in your Pulumi programs
Configuration
The defaultvalues.yaml for this template:
GVC and Locations
The GVC name and location list live underglobal.gvc so that Helm passes the same values to the bundled database and, through it, to etcd. Never configure the location lists separately.
global.gvc.name— Name of the GVC this chart creates. It must not already exist.global.gvc.locations[].name— A Control Plane location (e.g.aws-us-east-1). At least 2 are required, and the chart refuses to render with fewer.global.gvc.locations[].replicas— Database members in that location, not Grafana instances. It must be at least 1; removing a location from the list is the supported way to shrink the deployment.
Grafana UI Tier
image— The Grafana OSS container image. An Alpine-based variant is required: the boot wrapper needs a shell.replicas— Grafana UI instances per location. It carries no alerting-related restriction, because alert evaluation is a separate workload — the only thing to watch is the connection budget below.resources—minCpu/minMemoryare the reservation,maxCpu/maxMemorythe limit, applied per instance.database.maxOpenConn— Maximum database connections per instance.
Admin Credentials and Encryption Key
Both credentials live in opaque secrets you create before installing (see Prerequisites) — neither ever passes through Helm values. They sit next to each other invalues.yaml but their lifecycles are opposites:
Both workloads carry the admin bootstrap environment deliberately. Grafana’s built-in default password is the literal string
admin, so an evaluator without it that won the race for the empty database would create an admin/admin account on a publicly exposed UI. Testing confirms that does not happen: admin/admin is refused with a 401 on a fresh install.
Datasources as Code
datasources.definitions entries are standard Grafana datasource provisioning entries, rendered verbatim into a provisioning file that every instance of both workloads mounts at boot. Applying the same file concurrently from every instance is safe — testing at ten instances produced exactly one row per definition.
Credentials never go into the provisioning file. Put them in a dictionary secret created before installing, list it under datasources.credentialSecrets, and reference each key as $KEY:
reveal on exactly that secret. Provisioned datasources are read-only in the UI — change the value and upgrade to change one.
SMTP
Grafana sends alert notification emails through the SMTP server you configure, and the alert evaluator is what actually sends them.smtp.passwordSecretName is required whenever smtp.user is set, and the password stays in a pre-created opaque secret with encoding plain. Leave smtp.enabled: false if you use webhook or chat contact points instead.
Access
publicAccess.enabled applies to the UI tier only. When it is on, the tier is served on the canonical *.cpln.app HTTPS endpoint and Grafana’s root_url is derived from it automatically, including for the evaluator, so links in delivered notifications point at the UI tier and open in a browser. The evaluator itself is never reachable from the internet — it gets a canonical endpoint but requests to it are refused with a 403.
A firewall change is not instant. Turning public access off was measured at about 107 seconds after the new workload version was serving. Allow up to about four minutes and re-test before concluding a setting did not apply.
App Database
ThepostgresML block configures the bundled postgres-multi-location cluster: one primary, asynchronous replicas in the other locations, an HAProxy leader-routing tier in each, and automatic promotion in a surviving location. Everything the database tier can do — pooling, restores, emergency quorum recovery, per-member addressing — is documented on that template’s page.
postgresML.primaryLocation is a preferred location for the primary: the members elsewhere wait up to 90 seconds for it to initialize the cluster before bootstrapping themselves, and it also biases later elections. Keep it aligned with alerting.location so the evaluator’s queries have no cross-region hop.
Alert Evaluation
Grafana’s memberlist alerting HA coordinates instances over a UDP gossip channel, and UDP is not available between workloads on Control Plane — so if every instance evaluated rules you would get one notification per instance. Grafana also supports a Redis-backed alerting HA path, which would work here; it is deferred to a later version because it needs a stretched Redis tier, not because alerting HA is impossible on the platform. This template makes exactly-once evaluation a property of the topology instead: rule execution is disabled on the UI tier and enabled on a separate workload pinned to one replica inalerting.location, with zero replicas in every other location. Nothing is elected at runtime.
Verified in testing at replicas: 3 across three locations: all 9 UI instances report rule execution disabled, the single evaluator reports it enabled, and the other two locations run no evaluator replica at all. Behaviorally, an always-firing rule produced 8 notifications with 8 distinct request IDs over the observation window, every one from the evaluator’s location, with never two in the same instant.
Four consequences worth knowing before you rely on it:
-
Losing the evaluator’s replica self-heals. The platform reschedules it and evaluation resumes with no operator action — a measured gap of 21 seconds between the last notification from the old replica and the first from its replacement. The UI tier was unaffected throughout (every probe returned
200). -
Losing that whole location does not self-heal. Alert evaluation stops until you run
helm upgradewithalerting.locationset to a surviving location, and the UI gives no sign of it — dashboards look perfectly healthy while nothing is being evaluated. -
Moving
alerting.locationdoes not open a two-evaluator window. Only the evaluator workload is updated; the UI tier is untouched and does not restart. In the measured relocation the old evaluator’s last notification preceded the new evaluator’s readiness by 57 seconds, so the two never overlapped. Expect a gap of a few minutes with no evaluation while the new one boots. -
Silences do not propagate between instances. Without gossip, a silence created against the UI tier is not guaranteed to be honored by the evaluator, so create silences against the evaluator directly:
alerting.location: "" renders no evaluator at all — rules can still be created and viewed, and are never evaluated.
Cross-Location Behavior
Every instance shares one database, so there is no replication step between a write in one location and a read in another. Verified across three locations atreplicas: 2 — six UI instances plus the evaluator, each result attributed to a named individual instance:
Cross-region visibility was bounded at roughly 1–3 seconds by the measurement resolution rather than by the system.
The public endpoint is proximity-routed, not round-robin. In a 100-request tabulation, all 100 requests from one client were served by a single location — the one nearest the client. Several locations give you regional redundancy and locally served traffic, not a request spread across regions.
Connecting
Substitute your release name and the GVC name fromglobal.gvc.name.
Health and readiness are served at
/api/health, which reports the Grafana version and the app-database status. The Grafana HTTP API is available on the same endpoint for scripted dashboard, datasource and alert-rule management.
An in-GVC request to
RELEASE_NAME-grafana-ml.GVC_NAME.cpln.local:3000 is always served by an instance in the caller’s own location. That is why a single write-then-read from one client proves nothing about cross-location state.Availability and Planned Outages
Measured on three locations (aws-us-east-1, aws-eu-central-1, aws-us-west-2).
A single
Failed to lock database line during a cold start is expected. Grafana takes a non-blocking advisory lock to run schema migrations and exits if another instance holds it; the platform restarts it and the next attempt succeeds against the already-migrated schema. Seven restarts across the tier were observed on a three-location cold install, all from this race.Backing Up
Backups are disabled by default and cover the app database — the dashboards, users, alert rules and saved datasources that make up your Grafana. Enable them withpostgresML.backup.enabled: true, choose logical (a scheduled pg_dumpall cron workload in the one location named by postgresML.backup.location) or wal-g (continuous archiving from whichever member is currently the primary), and complete the storage setup for your provider before installing.
- AWS S3
- Google Cloud Storage
- MinIO / S3-compatible
1
Create a bucket
Create an S3 bucket. Set
postgresML.backup.aws.bucket and postgresML.backup.aws.region to match.2
Set up a Cloud Account
If you do not have one, create a Cloud Account for the AWS account holding the bucket. Set
postgresML.backup.aws.cloudAccountName to its name.3
Create a bucket-scoped IAM policy
Create an IAM policy with the JSON below (replace
YOUR_BUCKET_NAME), then set postgresML.backup.aws.policyName to the policy’s name. This bucket-scoped policy is all the identity needs — no broad managed policy is required.4
Choose a prefix
Set
postgresML.backup.aws.prefix to the folder path within the bucket.Important Notes
- Create the admin password, encryption key and database credentials secrets before installing. The chart creates none of them; without them the deployment waits on secrets that do not exist while
helm installreports success. - The GVC in
global.gvc.namemust not already exist. Helm adopts an existing one and deletes it on uninstall, taking every unrelated workload with it. - Never rotate or delete the encryption key. Every instance in every location decrypts stored datasource credentials with it; changing it makes them all unreadable and alert rules that query them fail.
- Every
helm upgradethat touches the database tier is a planned outage. Database writes stop for about 117 s and the Grafana tiers return errors for longer — measured recovery about 4 minutes, one location 5–6 minutes. The first upgrade after any install costs this even for a Grafana-only change. Changes confined to the Grafana workloads roll with zero downtime. - An upgrade that adds a secret reference can pause the rollout for about 9–10 minutes while Helm reports success. It clears itself — do not re-run the upgrade or edit policies.
- Alert evaluation stops if you lose
alerting.location, and the UI will not show it. Repoint the knob and upgrade; that restarts only the evaluator. - Silences must be created against the evaluator, from a workload in
alerting.location— its internal address returns503from every other location. - The public endpoint is proximity-routed. More locations means regional redundancy and locally served traffic, not requests spread across regions.
- Every location except the database primary’s pays a cross-region round trip per query, because Grafana has no read/write splitting. Set
postgresML.primaryLocationwhere most of your users are. - Scaling
replicashas no alerting-related restriction — it applies to the UI tier only, in every location includingalerting.location. Watch the connection budget instead. replicas: 2or higher lengthens a cold install to roughly 7 minutes per location and up to about 12 minutes for every replica. Nothing is wrong; there is simply more to schedule.- Grafana Live has no HA engine here, so a live-streamed message reaches only the browsers connected to the same instance. Dashboard auto-refresh, queries, alerting, provisioning, login and the API are unaffected.
- With
publicAccess.enabled: false, links in alert notifications point at the internal GVC address and will not open from a browser outside the GVC. - Never suspend a location. Suspending and resuming one permanently withdraws its endpoints from the other locations’ service discovery while every status surface still reports healthy. Remove a location from
global.gvc.locationsinstead. - Uninstall deletes the database volume sets along with the GVC — enable backups if the data matters. Prerequisite secrets you created are not owned by the release and survive it.
- This template ships Grafana OSS only — Enterprise features such as fine-grained RBAC, reporting and query caching are not available.
External References
Grafana Documentation
Official Grafana documentation
Grafana Alerting
Alert rules, contact points, notification policies, and silences
Alerting High Availability
How Grafana coordinates alert evaluation across instances
Provisioning Datasources
Reference for the datasource provisioning entries used by this template
Configuration Reference
Every Grafana setting and its environment-variable name
Grafana HTTP API
Manage dashboards, datasources, and alert rules programmatically
Grafana Multi-Location Template
View the source files, default values, and chart definition