> ## Documentation Index > Fetch the complete documentation index at: https://docs.controlplane.com/llms.txt > Use this file to discover all available pages before exploring further. # Grafana Multi-Location > Deploy Grafana OSS across several Control Plane locations behind one endpoint, with dashboards, users, sessions and datasource credentials shared through a stretched Patroni PostgreSQL cluster, and alert rules evaluated exactly once by a single pinned evaluator. ## Overview Grafana Multi-Location deploys **one logical Grafana** whose UI/API instances run in every location of a chart-created multi-location GVC, behind a single georouted `*.cpln.app` endpoint. Every dashboard, user, org, session, alert rule and saved datasource lives in a [`postgres-multi-location`](/template-catalog/templates/postgres-multi-location) cluster stretched across the same locations, so Grafana itself holds no state: there is no volume, no session affinity and nothing to hand over when an instance is replaced. Alert rules are evaluated by a **separate single-replica workload** that runs in exactly one location with rule execution enabled, while it is disabled on every UI instance. Exactly-once evaluation is therefore a property of the topology — nothing is elected at runtime, and no value of `replicas` can produce a second evaluator. See [Alert Evaluation](#alert-evaluation). For a Grafana inside a single location, use the [Grafana](/template-catalog/templates/grafana) template instead — it also offers multi-replica alerting HA coordinated by Redis Sentinel, which this template does not. ### How Many Locations You Need The bundled database's consensus store ([`etcd-multi-location`](/template-catalog/templates/etcd-multi-location)) commits a write only when a **majority** of its members agree, and it runs one member per location. That arithmetic decides what survives — the Grafana tier itself has no quorum of its own. | Locations | Location losses survived | What a user sees when one location is lost | | --------- | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **2** | **0** | Dashboards still render at times, but logins, saves and alert-state writes fail. The survivor holds current data and stays read-only until it is promoted by hand. | | **3** | **1** | Automatic database failover; the endpoint keeps serving from the two surviving locations. | | **5** | **2** | Survives losing **two** locations. | With N locations you survive `floor((N-1)/2)` losses, so an even count buys nothing over the odd count below it. Losing the location named in `alerting.location` is a separate matter — see [Alert Evaluation](#alert-evaluation). ### What Gets Created * **GVC** — A new GVC pinned to the configured locations. This template always creates it. * **Standard Grafana UI Workload** — `{release}-grafana-ml`, `replicas` instances **per location**, serving the UI and HTTP API on port `3000`. Public by default. Alert rule execution is disabled here. * **Standard Alert Evaluator Workload** — `{release}-grafana-ml-alerting`, the same image with **exactly one replica**, running only in `alerting.location` and never reachable from the internet. Optional — set `alerting.location: ""` to omit it entirely. * **App Database Workloads** — The `postgres-multi-location` subchart: a stateful Patroni PostgreSQL workload with one primary and asynchronous replicas, a stateful etcd workload for consensus, and an HAProxy leader-routing tier in every location. * **Volume Sets** — `{release}-postgres-ml-vs` for the PostgreSQL data directory and the etcd cluster's own volume set. Grafana has none. * **Secrets** — The subchart's startup scripts, and — only when `datasources.definitions` is set — `{release}-grafana-ml-datasources`, the rendered datasource provisioning file mounted by both Grafana workloads. The admin password, encryption key and database credentials are **prerequisite secrets you create yourself**; the chart references them by name and never creates, modifies or deletes them. * **Identity & Policy** — One identity shared by both Grafana workloads, with `reveal` on exactly the secrets they mount and nothing else. This template creates its own GVC. You do not need to create one before installing. ## Prerequisites The GVC named in `global.gvc.name` **must not already exist**. Helm adopts a GVC that does, and `helm uninstall` then deletes it along with every unrelated workload in it. Always point this template at a fresh GVC name. **Three secrets must exist before you install.** The admin account is a human-facing login and `publicAccess.enabled` defaults to `true`, so its password never passes through Helm values; the encryption key and the database credentials are shared by every instance in every location. An [opaque secret](/guides/create-secret/opaque) with encoding `plain`: ```bash theme={null} printf '%s' "$(openssl rand -hex 24)" \ | cpln secret create-opaque --name my-grafana-ml-admin-password --encoding plain -f - ``` Set `admin.passwordSecretName` to the name you used. An [opaque secret](/guides/create-secret/opaque) with encoding `plain`. Every instance reads it on every boot to decrypt datasource credentials stored in the shared database — back it up outside Control Plane and never rotate it: ```bash theme={null} printf '%s' "$(openssl rand -hex 32)" \ | cpln secret create-opaque --name my-grafana-ml-secret-key --encoding plain -f - ``` Set `admin.secretKeySecretName` to the name you used. A [dictionary secret](/guides/create-secret/dictionary) holding exactly `username`, `password` and `database`: ```bash theme={null} cpln secret create-dictionary --name my-grafana-ml-db-credentials \ --entry username=grafana \ --entry password="$(openssl rand -hex 24)" \ --entry database=grafana ``` Use plain identifiers for `username` and `database` — they are used unquoted when the database is created. Set `postgresML.postgres.credentialsSecretName` to the secret's name. ```bash theme={null} cpln secret reveal my-grafana-ml-db-credentials -o json ``` Without `-o json` the command prints a table containing no secret data. **A missing prerequisite secret wedges the install rather than failing it.** `cpln helm install` still reports success while the affected workload sits at zero replicas waiting on a secret reference that never resolves, which looks like a broken install. Create all three first, and confirm with `cpln workload get-deployments {release}-grafana-ml --gvc {gvc}` rather than trusting the Helm output. Optional features each need something created **before** you install: | Feature | What you must create first | | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------ | | Credentialed provisioned datasources | One [dictionary secret](/guides/create-secret/dictionary) per entry in `datasources.credentialSecrets`, holding the credential keys | | Authenticated SMTP | An [opaque secret](/guides/create-secret/opaque) with encoding `plain` holding the SMTP password, named in `smtp.passwordSecretName` | | Database backups | A bucket and access setup on AWS S3, Google Cloud Storage, or an S3-compatible server — see [Backing Up](#backing-up) | Once the secrets exist, install the template using your preferred method: Browse, install, and manage templates visually Manage templates from your terminal }> Declare templates in your Terraform configurations Pulumi Icon Streamline Icon: https://streamlinehq.com } > Declare templates in your Pulumi programs ## Configuration The default `values.yaml` for this template: ```yaml theme={null} # ─── GVC and locations ──────────────────────────────────────────────────────── # Lives under `global` so the postgres-multi-location subchart — and its own etcd # subchart, two levels down — get the same GVC and locations automatically. # Never maintain two location lists. global: gvc: # This chart CREATES this GVC. It must NOT already exist: Helm adopts a GVC # that does, and `helm uninstall` then DELETES it and everything in it. name: grafana-multi-location-gvc # Minimum 2 locations. The database tier needs 3 for automatic failover and # 5 to survive losing two — see the survival table in the README. # `replicas` here is DATABASE members per location. Grafana's own count per # location is the top-level `replicas` below. locations: - name: aws-us-east-1 replicas: 1 - name: aws-eu-central-1 replicas: 1 - name: aws-us-west-2 replicas: 1 # ─── Grafana UI tier ────────────────────────────────────────────────────────── image: grafana/grafana:13.1.3 replicas: 1 # Grafana UI instances PER LOCATION; scale freely, alerting is a separate workload resources: maxCpu: 1000m maxMemory: 1Gi minCpu: 500m minMemory: 512Mi database: maxOpenConn: 10 # per instance; (replicas × locations + 1) × this must stay under 80 # ─── Alerting ───────────────────────────────────────────────────────────────── # Alert rules are evaluated by a SEPARATE single-replica workload that runs only # in this location and is never exposed publicly. Nothing dedupes two evaluators # here: Grafana's memberlist alerting HA gossips over UDP, which is unavailable # between workloads on this platform. (Grafana also supports a Redis-backed # alerting HA path, which WOULD work here — it is deferred to a later version # because it needs a stretched Redis tier, not because it is impossible.) # A replica failure self-heals; losing this whole LOCATION stops evaluation until # you point this elsewhere and upgrade. The UI tier is unaffected either way. alerting: location: aws-us-east-1 # must be one of global.gvc.locations; "" = no evaluator workload, alerting disabled resources: maxCpu: 1000m maxMemory: 1Gi minCpu: 500m minMemory: 512Mi # ─── Admin Bootstrap & Encryption (prerequisite secrets) ────────────────────── # BOTH secrets must EXIST BEFORE INSTALL — opaque, encoding: plain (see README). # The admin login is on the public internet when publicAccess.enabled is true, # so its password never transits values or the Helm release. admin: user: admin # initial admin login name (not sensitive) # The password applies only when the admin account is FIRST created; on later # boots Grafana ignores it (change it in the UI instead). applyPassword: true # set false after your first login to stop referencing the password secret passwordSecretName: my-grafana-ml-admin-password # opaque secret holding the first-boot admin password # DIFFERENT LIFECYCLE — permanent. Read on EVERY boot by EVERY instance to # decrypt datasource credentials stored in the shared database: never delete # it, never rotate it. secretKeySecretName: my-grafana-ml-secret-key # opaque secret holding the encryption key # ─── Datasources as Code (optional) ─────────────────────────────────────────── # Grafana datasource provisioning entries, passed through verbatim. Every # instance of both workloads applies the same file on boot. datasources: definitions: [] # - name: Prometheus # type: prometheus # access: proxy # url: http://RELEASE-prometheus.GVC.cpln.local:9095 # isDefault: true # - name: AppDB # type: postgres # url: my-db-host:5432 # user: grafana_reader # jsonData: { database: appdb, sslmode: disable } # secureJsonData: # password: $PG_PASSWORD # interpolated from credentialSecrets below # The credentials Grafana authenticates TO each datasource with. `definitions` # renders into a plaintext provisioning file, so put the password in a # pre-created dictionary secret and write $KEY above. Every $KEY needs an entry # here, and the secrets MUST EXIST BEFORE INSTALL. credentialSecrets: [] # - name: my-grafana-ml-ds-credentials # keys: [PG_PASSWORD] # ─── SMTP for Alert Emails (optional) ───────────────────────────────────────── # The alerting workload is what actually sends. smtp: enabled: false host: smtp.example.com:587 # host:port user: "" # empty = unauthenticated SMTP. IF YOU SET THIS, the relay MUST offer # STARTTLS or TLS: Grafana refuses to send credentials over an unencrypted # connection ("failed to send email: unencrypted connection") and every # notification is then lost, with the only signal a log line in the # alerting workload. Hosted relays (SES, SendGrid, Mailgun, M365, Gmail) # are fine; a plain in-GVC relay is not — leave this empty for those. passwordSecretName: "" # opaque secret (encoding: plain) with the SMTP password; create BEFORE install fromAddress: grafana@example.com fromName: Grafana # ─── Access ─────────────────────────────────────────────────────────────────── # Applies to the UI tier. The alerting workload is NEVER public; it honours # internalAccess only, so you can reach it in-GVC to manage silences. publicAccess: enabled: true # UI on the canonical *.cpln.app HTTPS endpoint internalAccess: type: same-gvc # options: none, same-gvc, same-org, workload-list workloads: [] # used with workload-list # workloads: # - //gvc/GVC_NAME/workload/WORKLOAD_NAME # ─── App Database (subchart: postgres-multi-location) ───────────────────────── # One Patroni cluster stretched across the same locations: a single primary, # async replicas elsewhere, automatic promotion in a surviving location. # Grafana has no read/write splitting, so EVERY query goes to the primary — # put it where most of your users are. postgresML: postgres: # REQUIRED PREREQUISITE SECRET — CREATE IT BEFORE YOU INSTALL. # A `dictionary` secret holding exactly `username`, `password` and # `database`. If it does not exist the deployment wedges waiting on it. credentialsSecretName: my-grafana-ml-db-credentials # Preferred location for the database primary. Keep it aligned with # alerting.location so the hot path has no cross-region hop. primaryLocation: aws-us-east-1 resources: minCpu: 500m minMemory: 1Gi maxCpu: 1 maxMemory: 2Gi volumeset: capacity: 10 # initial capacity in GiB per member (minimum is 10) proxy: minReplicas: 2 # HAProxy leader-routing tier, per location maxReplicas: 2 backup: # optional database backups — see Storage setup in the README enabled: false mode: logical # logical or wal-g location: aws-us-east-1 # logical mode only: the ONE location the nightly job runs in resources: cpu: 100m # 512Mi, matching postgres-multi-location's own default. At 128Mi the GCP # path OOMs with NO log output: logical jobs merely report `failed` and the # wal-g sidecar loops on OOMKilled while WAL archives with no base backup. # Do not lower this without re-testing the GCS path. memory: 512Mi logical: image: ghcr.io/controlplane-com/backup-images/postgres-backup:17.1.0 schedule: "0 2 * * *" walg: intervalSeconds: 21600 provider: aws # options: aws, gcp, minio aws: bucket: my-grafana-ml-bucket region: us-east-1 cloudAccountName: my-s3-cloud-account policyName: my-grafana-ml-backup-policy prefix: grafana/backups gcp: bucket: my-grafana-ml-bucket cloudAccountName: my-gcs-cloud-account prefix: grafana/backups minio: endpoint: http://my-minio-workload:9000 bucket: my-grafana-ml-bucket credentialsSecretName: my-grafana-ml-minio-credentials prefix: grafana/backups ``` ### GVC and Locations The GVC name and location list live under `global.gvc` so that Helm passes the same values to the bundled database and, through it, to etcd. Never configure the location lists separately. * `global.gvc.name` — Name of the GVC this chart creates. It must not already exist. * `global.gvc.locations[].name` — A Control Plane location (e.g. `aws-us-east-1`). At least 2 are required, and the chart refuses to render with fewer. * `global.gvc.locations[].replicas` — **Database** members in that location, not Grafana instances. It must be at least 1; removing a location from the list is the supported way to shrink the deployment. ### Grafana UI Tier * `image` — The Grafana OSS container image. An Alpine-based variant is required: the boot wrapper needs a shell. * `replicas` — Grafana UI instances **per location**. It carries no alerting-related restriction, because alert evaluation is a separate workload — the only thing to watch is the connection budget below. * `resources` — `minCpu`/`minMemory` are the reservation, `maxCpu`/`maxMemory` the limit, applied per instance. * `database.maxOpenConn` — Maximum database connections **per instance**. `(replicas × locations + 1) × database.maxOpenConn` must stay at **80 or less** — the bundled cluster's `max_connections` is 100, and the remainder is headroom for Patroni and administration. The `+ 1` is the alert evaluator. The chart enforces the budget at render time and refuses to install with the arithmetic spelled out, so raising `replicas` past that point means lowering `database.maxOpenConn` (or lowering `replicas`). Signup, anonymous access and upstream analytics are disabled by the template and are not configurable. ### Admin Credentials and Encryption Key Both credentials live in [opaque secrets](/guides/create-secret/opaque) you create before installing (see [Prerequisites](#prerequisites)) — neither ever passes through Helm values. They sit next to each other in `values.yaml` but their lifecycles are opposites: | | `admin.passwordSecretName` | `admin.secretKeySecretName` | | ---------------------------- | -------------------------------------------------------------------- | ------------------------------------------------ | | What it holds | The `admin` login password | The key encrypting stored datasource credentials | | When Grafana reads it | Only when the admin account is **first created** | On **every boot**, on every instance | | Can you stop referencing it? | Yes — set `admin.applyPassword: false` | **No.** There is no toggle | | Can you rotate or delete it? | Yes, once `applyPassword` is `false` — change the password in the UI | **Never** | Both workloads carry the admin bootstrap environment deliberately. Grafana's built-in default password is the literal string `admin`, so an evaluator without it that won the race for the empty database would create an `admin`/`admin` account on a publicly exposed UI. Testing confirms that does not happen: `admin`/`admin` is refused with a `401` on a fresh install. The encryption key is permanent. Every instance in every location decrypts saved datasource credentials with it, so changing its payload makes all of them unreadable everywhere — and alert rules that query those datasources then fail. Back the key up outside Control Plane instead of rotating it. ### Datasources as Code `datasources.definitions` entries are standard [Grafana datasource provisioning](https://grafana.com/docs/grafana/latest/administration/provisioning/#data-sources) entries, rendered verbatim into a provisioning file that every instance of both workloads mounts at boot. Applying the same file concurrently from every instance is safe — testing at ten instances produced exactly one row per definition. Credentials never go into the provisioning file. Put them in a [dictionary secret](/guides/create-secret/dictionary) created before installing, list it under `datasources.credentialSecrets`, and reference each key as `$KEY`: ```yaml theme={null} datasources: definitions: - name: AppDB type: postgres url: my-db-host:5432 user: grafana_reader jsonData: { database: appdb, sslmode: disable } secureJsonData: password: $PG_PASSWORD credentialSecrets: - name: my-grafana-ml-ds-credentials keys: [PG_PASSWORD] ``` Each listed key is exposed to both workloads as an environment variable and interpolated when Grafana loads the file; the chart grants the identity `reveal` on exactly that secret. Provisioned datasources are read-only in the UI — change the value and upgrade to change one. ### SMTP **Authenticated SMTP requires a relay that offers STARTTLS or TLS.** Grafana refuses to send credentials over an unencrypted connection — it fails with `unencrypted connection` and **every notification is lost**, with the only signal a log line in the alerting workload, which is not publicly reachable. Hosted relays (SES, SendGrid, Mailgun, Microsoft 365, Gmail) are unaffected; a plain in-GVC relay is not. Leave `smtp.user` empty to send unauthenticated against such a relay. Grafana sends alert notification emails through the SMTP server you configure, and the **alert evaluator is what actually sends them**. `smtp.passwordSecretName` is required whenever `smtp.user` is set, and the password stays in a pre-created [opaque secret](/guides/create-secret/opaque) with encoding `plain`. Leave `smtp.enabled: false` if you use webhook or chat contact points instead. ### Access `publicAccess.enabled` applies to the **UI tier only**. When it is on, the tier is served on the canonical `*.cpln.app` HTTPS endpoint and Grafana's `root_url` is derived from it automatically, including for the evaluator, so links in delivered notifications point at the UI tier and open in a browser. The evaluator itself is never reachable from the internet — it gets a canonical endpoint but requests to it are refused with a `403`. | `internalAccess.type` | Description | | --------------------- | ------------------------------------------------------------------------ | | `none` | No internal access. | | `same-gvc` | Allow access from all workloads in the same GVC (default). | | `same-org` | Allow access from all workloads in the same organization. | | `workload-list` | Allow access only from the workload links in `internalAccess.workloads`. | A firewall change is not instant. Turning public access off was measured at about 107 seconds after the new workload version was serving. Allow up to about four minutes and re-test before concluding a setting did not apply. ### App Database The `postgresML` block configures the bundled [`postgres-multi-location`](/template-catalog/templates/postgres-multi-location) cluster: one primary, asynchronous replicas in the other locations, an HAProxy leader-routing tier in each, and automatic promotion in a surviving location. Everything the database tier can do — pooling, restores, emergency quorum recovery, per-member addressing — is documented on that template's page. `postgresML.primaryLocation` is a **preferred** location for the primary: the members elsewhere wait up to 90 seconds for it to initialize the cluster before bootstrapping themselves, and it also biases later elections. Keep it aligned with `alerting.location` so the evaluator's queries have no cross-region hop. Grafana has **no read/write splitting** — every query, including every dashboard load, goes to the single primary. Testing confirms the shape directly: with seven Grafana instances running, all application connections were on the primary and both standbys carried none. Every location except the primary's therefore pays one cross-region round trip per query, so set `primaryLocation` where most of your users are. ## Alert Evaluation Grafana's memberlist alerting HA coordinates instances over a UDP gossip channel, and UDP is not available between workloads on Control Plane — so if every instance evaluated rules you would get one notification per instance. Grafana also supports a **Redis-backed** alerting HA path, which would work here; it is deferred to a later version because it needs a stretched Redis tier, not because alerting HA is impossible on the platform. This template makes exactly-once evaluation a property of the **topology** instead: rule execution is disabled on the UI tier and enabled on a separate workload pinned to one replica in `alerting.location`, with zero replicas in every other location. Nothing is elected at runtime. Verified in testing at `replicas: 3` across three locations: all **9 UI instances** report rule execution disabled, the **single evaluator** reports it enabled, and the other two locations run no evaluator replica at all. Behaviorally, an always-firing rule produced **8 notifications with 8 distinct request IDs** over the observation window, every one from the evaluator's location, with never two in the same instant. Four consequences worth knowing before you rely on it: * **Losing the evaluator's replica self-heals.** The platform reschedules it and evaluation resumes with no operator action — a measured gap of **21 seconds** between the last notification from the old replica and the first from its replacement. The UI tier was unaffected throughout (every probe returned `200`). * **Losing that whole location does not self-heal.** Alert evaluation stops until you run `helm upgrade` with `alerting.location` set to a surviving location, and the UI gives no sign of it — dashboards look perfectly healthy while nothing is being evaluated. * **Moving `alerting.location` does not open a two-evaluator window.** Only the evaluator workload is updated; the UI tier is untouched and does not restart. In the measured relocation the old evaluator's last notification preceded the new evaluator's readiness by **57 seconds**, so the two never overlapped. Expect a gap of a few minutes with no evaluation while the new one boots. * **Silences do not propagate between instances.** Without gossip, a silence created against the UI tier is not guaranteed to be honored by the evaluator, so create silences against the evaluator directly: ```bash theme={null} curl http://RELEASE_NAME-grafana-ml-alerting.GVC_NAME.cpln.local:3000/api/alertmanager/grafana/api/v2/silences ``` **The evaluator only answers from inside `alerting.location`.** Its internal name resolves to the GVC address from every location, but the other locations have no local upstream for it and return `503`. Run the silence command from a workload replica that is running in `alerting.location`. Setting `alerting.location: ""` renders no evaluator at all — rules can still be created and viewed, and are never evaluated. ## Cross-Location Behavior Every instance shares one database, so there is no replication step between a write in one location and a read in another. Verified across three locations at `replicas: 2` — six UI instances plus the evaluator, each result attributed to a named individual instance: | What was tested | Result | | ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | | Dashboard created in one location | Read back **byte-identical** from all six UI instances and the evaluator, including the internal id | | Edit made in a second location | Authoritative everywhere, including the location it was created in — writes flow in both directions | | User and org created in one location | Log in successfully against every other instance (a wrong password returns `401` as the control) | | Session cookie issued in one location | Accepted as the only credential by every other instance, returning the correct identity — **no session affinity is needed** | | Two replicas in the same location | Each sees the other's writes | | Datasource credentials | Decrypt and connect successfully on **every** instance, because all of them share `GF_SECURITY_SECRET_KEY` | Cross-region visibility was bounded at roughly 1–3 seconds by the measurement resolution rather than by the system. **The public endpoint is proximity-routed, not round-robin.** In a 100-request tabulation, all 100 requests from one client were served by a single location — the one nearest the client. Several locations give you regional redundancy and locally served traffic, not a request spread across regions. ## Connecting Substitute your release name and the GVC name from `global.gvc.name`. | What | Where | | --------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | | Grafana UI / API (public) | `status.canonicalEndpoint` of `{release}-grafana-ml` — `cpln workload get RELEASE_NAME-grafana-ml --gvc GVC_NAME -o yaml` | | Grafana UI / API (internal) | `RELEASE_NAME-grafana-ml.GVC_NAME.cpln.local:3000` | | Alert evaluator (internal, own location only) | `RELEASE_NAME-grafana-ml-alerting.GVC_NAME.cpln.local:3000` | | App database (always the current primary) | `RELEASE_NAME-postgres-ml-proxy.GVC_NAME.cpln.local:5432` | | Admin login | `admin.user`, with the password in the secret named by `admin.passwordSecretName` — `cpln secret reveal SECRET_NAME -o json` | Health and readiness are served at `/api/health`, which reports the Grafana version and the app-database status. The [Grafana HTTP API](https://grafana.com/docs/grafana/latest/developers/http_api/) is available on the same endpoint for scripted dashboard, datasource and alert-rule management. An in-GVC request to `RELEASE_NAME-grafana-ml.GVC_NAME.cpln.local:3000` is always served by an instance in the **caller's own** location. That is why a single write-then-read from one client proves nothing about cross-location state. ## Availability and Planned Outages Measured on three locations (`aws-us-east-1`, `aws-eu-central-1`, `aws-us-west-2`). | Event | Measured impact | | ----------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Cold install to all tiers ready, `replicas: 1` | **4 m 44 s** — HAProxy and etcd first, then the Patroni cluster, then Grafana's 713 schema migrations (5.79 s with the primary local) | | Cold install at `replicas: 2` | About **7 minutes** for every location to report ready, and up to about **12 minutes** for every individual replica | | Grafana-only change (for example `replicas: 1` → `3`) | **Zero downtime** — 1160 probe samples across four endpoints, no failures. **Does not apply to the first upgrade after an install**, which re-applies every tier once regardless of what changed | | Any `helm upgrade` that touches the database tier | Database writes unavailable in **every** location for about **117 s**; the Grafana tier is down longer because reads fail too — measured recovery about **4 minutes**, with one location out for **5–6 minutes** | | Loss of the evaluator's replica | **21 s** without alert evaluation; the UI tier is unaffected | **Treat every `helm upgrade` as a planned outage rather than a rolling one.** The bundled database members do not restart one at a time — the field that would serialize the rollout is not retained by the platform, so they go down together. Grafana's readiness probe is `/api/health`, which reports the database, so while the database is down the whole Grafana tier drops out of the load balancer and returns `503`. Changes confined to the Grafana workloads do not have this cost. **An upgrade that adds a new secret reference can pause the rollout for about 9–10 minutes while `helm upgrade` reports success.** Affected locations show `The identity ... is not allowed to reveal the secret ...` even though the grant is already in place and visible in `cpln secret access-report`. It **clears itself with no action** (measured 9 m 0 s – 9 m 30 s); do not re-run the upgrade or start editing policies. Setting `admin.applyPassword` back to `true`, enabling `smtp` with a password, and adding `datasources.credentialSecrets` are the changes that trigger it. It also means Helm reporting success is not evidence the new version is running — check `cpln workload get-deployments`. **A single `Failed to lock database` line during a cold start is expected.** Grafana takes a non-blocking advisory lock to run schema migrations and exits if another instance holds it; the platform restarts it and the next attempt succeeds against the already-migrated schema. Seven restarts across the tier were observed on a three-location cold install, all from this race. ## Backing Up Backups are disabled by default and cover the app database — the dashboards, users, alert rules and saved datasources that make up your Grafana. Enable them with `postgresML.backup.enabled: true`, choose `logical` (a scheduled `pg_dumpall` cron workload in the one location named by `postgresML.backup.location`) or `wal-g` (continuous archiving from whichever member is currently the primary), and complete the storage setup for your provider **before** installing. The backup path has been exercised against **AWS S3** and **MinIO**, and a **wal-g restore has been verified end to end** (base backup plus WAL replay into an empty data directory, checksum-identical to source). The logical restore and the volume-set swap procedure have not been exercised. The `gcp` and `minio` providers are configured the same way but have not been exercised — validate them in a test install, and rehearse your restore procedure, before relying on either. Create an S3 bucket. Set `postgresML.backup.aws.bucket` and `postgresML.backup.aws.region` to match. If you do not have one, [create a Cloud Account](/guides/create-cloud-account) for the AWS account holding the bucket. Set `postgresML.backup.aws.cloudAccountName` to its name. Create an IAM policy with the JSON below (replace `YOUR_BUCKET_NAME`), then set `postgresML.backup.aws.policyName` to the policy's name. This bucket-scoped policy is all the identity needs — no broad managed policy is required. ```json theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListBucket", "s3:GetObjectVersion", "s3:DeleteObjectVersion" ], "Resource": [ "arn:aws:s3:::YOUR_BUCKET_NAME", "arn:aws:s3:::YOUR_BUCKET_NAME/*" ] } ] } ``` Set `postgresML.backup.aws.prefix` to the folder path within the bucket. Create a GCS bucket. Set `postgresML.backup.gcp.bucket` to its name. If you do not have one, [create a Cloud Account](/guides/create-cloud-account) for the GCP project. Set `postgresML.backup.gcp.cloudAccountName` to its name. Add the **Storage Admin** (`roles/storage.admin`) role to the service account associated with the Cloud Account. The chart additionally binds `roles/storage.objectAdmin` on exactly the bucket named in `postgresML.backup.gcp.bucket`. Set `postgresML.backup.gcp.prefix` to the folder path. Create the bucket on the server. Set `postgresML.backup.minio.bucket` to its name. No Cloud Account is needed. Set `postgresML.backup.minio.endpoint` to the S3 API address including the port. For the [MinIO](/template-catalog/templates/minio) template in the same GVC, that is `http://WORKLOAD_NAME:9000`. Create a [dictionary secret](/guides/create-secret/dictionary) and set `postgresML.backup.minio.credentialsSecretName` to its name, then set `postgresML.backup.minio.prefix` to the folder path: ```bash theme={null} cpln secret create-dictionary --name my-grafana-ml-minio-credentials \ --entry accessKey=MINIO_ACCESS_KEY \ --entry secretKey=MINIO_SECRET_KEY ``` ## Important Notes * **Create the admin password, encryption key and database credentials secrets before installing.** The chart creates none of them; without them the deployment waits on secrets that do not exist while `helm install` reports success. * **The GVC in `global.gvc.name` must not already exist.** Helm adopts an existing one and deletes it on uninstall, taking every unrelated workload with it. * **Never rotate or delete the encryption key.** Every instance in every location decrypts stored datasource credentials with it; changing it makes them all unreadable and alert rules that query them fail. * **Every `helm upgrade` that touches the database tier is a planned outage.** Database writes stop for about 117 s and the Grafana tiers return errors for longer — measured recovery about 4 minutes, one location 5–6 minutes. The first upgrade after any install costs this even for a Grafana-only change. Changes confined to the Grafana workloads roll with zero downtime. * **An upgrade that adds a secret reference can pause the rollout for about 9–10 minutes** while Helm reports success. It clears itself — do not re-run the upgrade or edit policies. * **Alert evaluation stops if you lose `alerting.location`, and the UI will not show it.** Repoint the knob and upgrade; that restarts only the evaluator. * **Silences must be created against the evaluator, from a workload in `alerting.location`** — its internal address returns `503` from every other location. * **The public endpoint is proximity-routed.** More locations means regional redundancy and locally served traffic, not requests spread across regions. * **Every location except the database primary's pays a cross-region round trip per query**, because Grafana has no read/write splitting. Set `postgresML.primaryLocation` where most of your users are. * **Scaling `replicas` has no alerting-related restriction** — it applies to the UI tier only, in every location including `alerting.location`. Watch the connection budget instead. * **`replicas: 2` or higher lengthens a cold install** to roughly 7 minutes per location and up to about 12 minutes for every replica. Nothing is wrong; there is simply more to schedule. * **Grafana Live has no HA engine here**, so a live-streamed message reaches only the browsers connected to the same instance. Dashboard auto-refresh, queries, alerting, provisioning, login and the API are unaffected. * **With `publicAccess.enabled: false`, links in alert notifications point at the internal GVC address** and will not open from a browser outside the GVC. * **Never suspend a location.** Suspending and resuming one permanently withdraws its endpoints from the other locations' service discovery while every status surface still reports healthy. Remove a location from `global.gvc.locations` instead. * **Uninstall deletes the database volume sets** along with the GVC — enable backups if the data matters. Prerequisite secrets you created are not owned by the release and survive it. * **This template ships Grafana OSS only** — Enterprise features such as fine-grained RBAC, reporting and query caching are not available. ## External References Official Grafana documentation Alert rules, contact points, notification policies, and silences How Grafana coordinates alert evaluation across instances Reference for the datasource provisioning entries used by this template Every Grafana setting and its environment-variable name Manage dashboards, datasources, and alert rules programmatically View the source files, default values, and chart definition