Thanos

Overview

Thanos turns many independent Prometheus servers into one queryable metrics system. This template deploys Thanos Query — a stateless global PromQL layer that fans each query out over the gRPC Store API endpoints you list (typically the Thanos sidecar of each Prometheus template install, in any GVC or region) and deduplicates HA pairs — plus an optional object-storage tier (Store Gateway and Compactor) that serves and maintains long-term metric history in your bucket. This is a self-hosted query layer for your own metrics from your own Prometheus servers. It is separate from — and not a replacement for — Control Plane’s built-in observability, which keeps collecting and dashboarding your workloads’ metrics natively.

This template deploys into a GVC you already have — it does not create, provision or manage one. Every workload below runs in each location of that GVC, so the Compactor (a strict one-per-bucket singleton) belongs in a GVC with exactly one location — see Long-Term Storage Tier.

What Gets Created

Resource	Name	Purpose	Optional
Workload	`RELEASE_NAME-thanos`	Thanos Query: UI and PromQL API on HTTP port `10902`, its own Store API on gRPC port `10901` (standard, `replicas` instances)	—
Workload	`RELEASE_NAME-thanos-store`	Store Gateway: serves historical blocks from your bucket to Query over gRPC `10901` (stateful, 1 replica)	`storeGateway.enabled: true`
Workload	`RELEASE_NAME-thanos-compact`	Compactor: compacts, downsamples and applies retention to the blocks in your bucket (stateful, 1 replica, no internal inbound)	`compactor.enabled: true`
Volume set	`RELEASE_NAME-thanos-store-vs`	Store Gateway local index and metadata cache, 10 GiB by default — safe to lose, it rebuilds	`storeGateway.enabled: true`
Volume set	`RELEASE_NAME-thanos-compact-vs`	Compactor workspace, 20 GiB by default	`compactor.enabled: true`
Secret	`RELEASE_NAME-thanos-objstore`	Rendered `objstore.yml` bucket configuration, mounted as a file by the Store Gateway and Compactor	`storeGateway.enabled` or `compactor.enabled`
Identity	`RELEASE_NAME-thanos-identity`	Shared by all workloads; carries the AWS or GCP cloud access to your bucket when the storage tier is on	—
Policy	`RELEASE_NAME-thanos-policy`	`reveal` on the objstore secret and, with `storage.type: minio`, on your MinIO credentials secret	`storeGateway.enabled` or `compactor.enabled`

The MinIO credentials secret is a prerequisite you create yourself; the chart references it by name and never creates, modifies or deletes it.

Prerequisites

None for a default install. Query starts with an empty store list, and you point it at your Store API endpoints with stores (see Store Endpoints). The optional storage tier (storeGateway.enabled / compactor.enabled) needs an existing bucket in one of the supported backends — the same bucket your Prometheus template’s Thanos sidecar uploads blocks to. Complete the steps for your chosen backend, then set storage.type to match.

AWS S3
Google Cloud Storage
S3-compatible (MinIO, R2, Wasabi)

AWS S3 access is keyless: the workload identity receives temporary credentials through a Control Plane cloud account, and nothing is stored.

Create a bucket

Create an S3 bucket. Set storage.aws.bucket and storage.aws.region to match.

Create a bucket-scoped IAM policy

In AWS IAM, create a policy with the JSON below, replacing YOUR_BUCKET with your bucket name:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
      "Resource": [
        "arn:aws:s3:::YOUR_BUCKET",
        "arn:aws:s3:::YOUR_BUCKET/*"
      ]
    }
  ]
}

Set storage.aws.policyName to the policy’s bare name (not its ARN).

Set up a cloud account

If you do not have one, create a Cloud Account for your AWS account. Set storage.aws.cloudAccountName to its name, and storage.type to aws (the default).

Google Cloud Storage access is keyless through a Control Plane cloud account; nothing is stored.

Create a bucket

Create a GCS bucket. Set storage.gcp.bucket to its name.

Set up a cloud account

If you do not have one, create a Cloud Account for your GCP project. Set storage.gcp.cloudAccountName to its name, and storage.type to gcp.

Bucket access is granted for you

There is no policy to author: the template binds the workload identity to roles/storage.objectAdmin on exactly that bucket.

An S3-compatible server cannot federate with a cloud account, so it uses static access keys — held in a secret you create, never in Helm values.

Create a bucket and access keys

On your server, create the bucket and an access key pair that can read and write it. Set storage.minio.bucket and storage.minio.region to match.

Store the keys in a dictionary secret

Create a dictionary secret holding exactly the keys accessKey and secretKey. Secrets are org-level, so no --gvc flag is involved:

cpln secret create-dictionary --name SECRET_NAME \
  --entry accessKey='YOUR-ACCESS-KEY' \
  --entry secretKey='YOUR-SECRET-KEY'

Set storage.minio.credentialsSecretName to this name. The keys reach the Store Gateway and Compactor as AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, and the chart grants the workload identity reveal on exactly this secret.

Set the endpoint

Set storage.minio.endpoint to the S3 API address as host:port with no scheme (for example minio.example.com:9000; use the fully qualified WORKLOAD_NAME.GVC_NAME.cpln.local:PORT form for a server inside Control Plane), set storage.minio.insecure: true when the endpoint is plain HTTP, and set storage.type to minio.

A missing credentials secret wedges the storage tier rather than failing the install. cpln helm install reports success while RELEASE_NAME-thanos-store and RELEASE_NAME-thanos-compact sit with no running containers and therefore no logs. The only place the missing secret is named is status.versions[].message:

cpln workload get-deployments RELEASE_NAME-thanos-store --gvc GVC_NAME -o yaml

Create the missing secret and the deployment recovers on its own after several minutes, or run cpln workload force-redeployment RELEASE_NAME-thanos-store --gvc GVC_NAME (and the same for RELEASE_NAME-thanos-compact) to skip the wait.

Installation

A default install needs no values — it deploys Query alone, into your GVC:

cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/thanos \
  --version 1.2.0 --gvc GVC_NAME

Add your stores list (and, for long-term history, the storage tier) in a values file passed with -f, either at install time or later with cpln helm upgrade.

UI

Browse, install, and manage templates visually

CLI

Manage templates from your terminal

Terraform

Declare templates in your Terraform configurations

Pulumi

Declare templates in your Pulumi programs

Configuration

Query

image: quay.io/thanos/thanos:v0.42.2

resources: # Thanos Query
  maxCpu: 500m
  maxMemory: 1Gi
  minCpu: 250m
  minMemory: 512Mi

replicas: 1 # Query is stateless — set 2+ for an HA query tier

image is shared by all three components. resources sizes the Query container; the Store Gateway and Compactor have their own resources blocks under their keys. Query is stateless, so replicas of 2 or more needs no coordination and gives an HA query tier — see Scaling and Availability.

Store Endpoints

stores: []
# stores:
#   - my-prometheus-prometheus.metrics-east.cpln.local:10901
#   - my-prometheus-b-prometheus.metrics-west.cpln.local:10901

queryReplicaLabels: # external label(s) marking HA duplicates to deduplicate
  - replica

Each stores entry is a gRPC Store API endpoint — typically the Thanos sidecar of a Prometheus template install — written as host:port with no scheme. The chart refuses to render an entry carrying a scheme or lacking a port. Address every endpoint by its fully qualified service name, whether it lives in the same GVC, another GVC or another region:

stores:
  - RELEASE_NAME-prometheus.GVC_NAME.cpln.local:10901

For a Prometheus template install, RELEASE_NAME and GVC_NAME are that install’s own release and GVC; the sidecar serves the Store API on 10901. Two requirements apply to endpoints in another GVC, both on the Prometheus side:

The target workload’s internal firewall must allow inbound from this Query workload — set its internalAccess.type to same-org, or to workload-list including //gvc/GVC_NAME/workload/RELEASE_NAME-thanos (this install’s names). A store reported as down by Query is almost always this firewall.
Internal traffic between locations (Query in one region, sidecars in another) incurs egress charges — co-locate the query tier with its stores where practical.

queryReplicaLabels names the external label (or labels) that mark HA duplicates: series identical except for this label are collapsed into one at query time. The label name must exactly match the external label your HA Prometheus pair sets through the Prometheus template’s externalLabels (for example replica: a and replica: b). Endpoints are passed to Query as arguments, so changing stores takes effect through cpln helm upgrade, which redeploys Query with the new list. Query keeps answering while some listed endpoints are down.

Access

publicAccess:
  enabled: false # true = Query UI/API at a *.cpln.app URL — it has NO built-in auth
internalAccess:
  type: same-gvc # options: none, same-gvc, same-org, workload-list
  workloads: [] # used with workload-list, e.g. //gvc/GVC/workload/NAME

publicAccess.enabled — Serves the Query UI and PromQL API on a *.cpln.app canonical endpoint. Query has no built-in authentication: anyone with the URL can run queries against everything it federates. Keep it off (the default) and reach the UI through cpln port-forward (see Connecting), or put your own authenticating proxy in front.
internalAccess.type — Internal firewall scope of the Query workload:

Type	Description
`none`	No internal access.
`same-gvc`	Allow access from all workloads in the same GVC (default).
`same-org`	Allow access from all workloads in the same org.
`workload-list`	Allow access only from workloads listed in `internalAccess.workloads`.

internalAccess.workloads entries take the form //gvc/GVC_NAME/workload/WORKLOAD_NAME. Allow several minutes after changing an access setting before concluding it did not apply.

Long-Term Storage Tier

storeGateway:
  enabled: false # serves historical bucket blocks to Query
  resources:
    maxCpu: 500m
    maxMemory: 1Gi
    minCpu: 250m
    minMemory: 512Mi
  volumeset:
    capacity: 10 # GiB — local index/meta cache; safe to lose (rebuilds on start)

compactor:
  enabled: false # compacts + downsamples bucket blocks — MUST be the ONLY compactor on the bucket, across all installs (runs one per GVC location: use a single-location GVC)
  resources:
    maxCpu: 500m
    maxMemory: 1Gi
    minCpu: 250m
    minMemory: 512Mi
  volumeset:
    capacity: 20 # GiB — compaction workspace; size ~2x two weeks of raw blocks
  retention: # per-resolution bucket retention; "0d" keeps forever
    raw: "0d"
    fiveMinutes: "0d"
    oneHour: "0d"

The storage tier is off by default. It serves metric history from the bucket your Prometheus sidecars upload blocks to, configured under Object Storage:

storeGateway.enabled — Runs the Store Gateway, which serves historical bucket blocks to Query. Query is wired to it automatically, so it needs no stores entry. Its volume set is only a local cache.
compactor.enabled — Runs the Compactor, which compacts and downsamples bucket blocks and applies compactor.retention per resolution (raw, fiveMinutes, oneHour). Each value is a duration such as 30d, 2w or 1y, and "0d" keeps data forever; the chart refuses any other form.

Long-term reads need both halves on the same bucket: a Prometheus sidecar uploading blocks to it, and storeGateway.enabled: true here to serve them.

Run exactly one Compactor per bucket, across every install and region. A second Compactor on the same bucket corrupts the block layout and needs manual repair. Enable compactor in one install only, and only in a GVC with a single location — the Compactor workload runs one replica in each location of its GVC.

Object Storage

storage:
  type: aws # options: aws, gcp, minio

  # AWS S3 — keyless via Control Plane cloud identity.
  aws:
    bucket: my-thanos-bucket # bucket must already exist
    region: us-east-1
    cloudAccountName: my-s3-cloud-account # Control Plane AWS cloud account
    policyName: my-thanos-s3-policy # custom IAM policy granting bucket access (bare name)

  # Google Cloud Storage — keyless via Control Plane cloud identity.
  gcp:
    bucket: my-thanos-bucket # bucket must already exist
    cloudAccountName: my-gcs-cloud-account # Control Plane GCP cloud account

  # Any S3-compatible server (MinIO, R2, Wasabi, …) — static keys + endpoint.
  minio:
    endpoint: my-minio:9000 # host:port, NO scheme — use the FQDN, e.g. my-minio.my-gvc.cpln.local:9000
    insecure: true # true when the endpoint is plain HTTP
    bucket: my-thanos-bucket # bucket must already exist
    region: us-east-1
    # REQUIRED PREREQUISITE SECRET when type is `minio` — a `dictionary` secret
    # holding exactly `accessKey` and `secretKey`. They reach the store and compact
    # workloads as AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY rather than being
    # written into objstore.yml, because a cpln:// reference inside a config file
    # is never resolved.
    credentialsSecretName: my-thanos-minio-credentials

Set storage.type to aws, gcp or minio and fill in that block; the others are ignored. Nothing here is read unless storeGateway.enabled or compactor.enabled is true. The setup for each backend is under Prerequisites.

Field	Backends	Description
`bucket`	all	Target bucket. Must already exist.
`region`	`aws`, `minio`	Bucket region.
`cloudAccountName`	`aws`, `gcp`	Control Plane cloud account granting bucket access.
`policyName`	`aws`	Bare name of your bucket-scoped IAM policy.
`endpoint`	`minio`	S3 API address as `host:port`, no scheme.
`insecure`	`minio`	`true` when the endpoint is plain HTTP.
`credentialsSecretName`	`minio`	Dictionary secret holding `accessKey` and `secretKey`; must exist before install.

Connecting

What	Address	Notes
Query UI and PromQL API (internal)	`http://RELEASE_NAME-thanos.GVC_NAME.cpln.local:10902`	Reachable per `internalAccess.type`
Grafana datasource	`http://RELEASE_NAME-thanos.GVC_NAME.cpln.local:10902`	Add it as a Prometheus-type datasource — Query speaks the Prometheus HTTP API
Query UI and PromQL API (public)	`status.canonicalEndpoint` of `RELEASE_NAME-thanos`	`cpln workload get RELEASE_NAME-thanos --gvc GVC_NAME -o yaml`; present while `publicAccess.enabled` is `true`. No authentication
Query’s own Store API	`RELEASE_NAME-thanos.GVC_NAME.cpln.local:10901`	gRPC, for a higher Thanos query tier
Credentials	None	Query has no login of its own

With the default private setup, open a tunnel to the Query workload and browse to http://localhost:10902 for the UI:

cpln port-forward RELEASE_NAME-thanos 10902:10902 --gvc GVC_NAME

With the tunnel open, check readiness and run a PromQL query against everything Query federates:

curl -s http://localhost:10902/-/ready
curl -s 'http://localhost:10902/api/v1/query?query=up'

Point Grafana at Query instead of at individual Prometheus servers to get one deduplicated global view; the Grafana template shows the datasource definition. The Query UI lists every configured store with its health and labels — the first place to look when a source is missing from results.

Operations

Upgrading from 1.0.0

Version 1.1.0 renamed the resource limits in all three resource blocks: resources.cpu and resources.memory became resources.maxCpu and resources.maxMemory, and the same rename applies under storeGateway.resources and compactor.resources. The old keys are not rejected — they are silently ignored, so a values file still carrying them installs with the default limits. Rename them before you upgrade. The MinIO change under Upgrading from 1.1.0 also applies.

Upgrading from 1.1.0

Version 1.2.0 moved the MinIO object-storage credentials out of Helm values. storage.minio.accessKey and storage.minio.accessSecret were removed and replaced by storage.minio.credentialsSecretName, which names a dictionary secret you create. Installs using storage.type: aws or gcp are unaffected. With storage.type: minio, carrying the old keys stops the upgrade at render, before anything is applied:

Error: execution error at (thanos/templates/workload-query.yaml:1:4): thanos: minio.accessKey and
minio.accessSecret were REMOVED — they are now a `dictionary` secret you create, named by
minio.credentialsSecretName, holding the keys `accessKey` and `secretKey`. Delete them from your values.

Create the credentials secret

Use the key pair the install already uses — the value of accessKey becomes the accessKey entry, and the value of accessSecret becomes the secretKey entry:

cpln secret create-dictionary --name SECRET_NAME \
  --entry accessKey='YOUR-ACCESS-KEY' \
  --entry secretKey='YOUR-SECRET-KEY'

Update your values

Delete storage.minio.accessKey and storage.minio.accessSecret, and set storage.minio.credentialsSecretName to the secret’s name.

Upgrade the release

cpln helm upgrade RELEASE_NAME oci://ghcr.io/controlplane-com/templates/thanos \
  --version 1.2.0 --gvc GVC_NAME \
  -f values.yaml

The values file must use the chart’s nesting, exactly as the blocks in Configuration show. A flat dotted key is accepted without error and silently changes nothing.

Scaling and Availability

Query is stateless and scales horizontally: set replicas to 2 or more for a query tier that keeps answering through rolling restarts and the loss of a replica. replicas is the fixed instance count in each location of the GVC.
Store Gateway runs as one instance. Its volume set is a cache: losing it costs a rebuild on the next start, never data.
Compactor can never scale — it is a strict singleton per bucket. Its replica count is fixed at one and is not a value.
For a query tier in several regions, install this template once per regional GVC, each with the same stores list. The instances need no coordination, but only one of them may enable the Compactor.

Troubleshooting

A store is reported as down in the Query UI

Cause: almost always the target’s internal firewall. A Prometheus sidecar in another GVC does not accept inbound traffic from this Query workload by default.Fix: on the Prometheus install, set internalAccess.type to same-org, or to workload-list including //gvc/GVC_NAME/workload/RELEASE_NAME-thanos, and upgrade that release. Allow several minutes for the change to apply. Also confirm the entry is the fully qualified WORKLOAD_NAME.GVC_NAME.cpln.local:10901 form.

Install or upgrade fails with stores entries must be host port

Cause: a stores entry carries a scheme or has no port. The chart stops at render with thanos: stores entries must be host:port with NO scheme or thanos: stores entries must be host:port (gRPC Store API, typically :10901).Fix: write each entry as host:port only — for example RELEASE_NAME-prometheus.GVC_NAME.cpln.local:10901, with no http:// or grpc:// prefix.

Duplicate series from an HA Prometheus pair

Cause: queryReplicaLabels does not match the external label the pair sets, so Query sees two distinct series.Fix: set queryReplicaLabels to exactly the label name used in the pair’s externalLabels (replica by default), and upgrade.

Older data is missing from query results

Cause: long-term reads need both halves on the same bucket. Either the Prometheus sidecar is not uploading blocks, or the Store Gateway is off or pointed at a different bucket.Fix: enable object storage on the Prometheus install, set storeGateway.enabled: true here, and make sure both name the same bucket and backend.

The Store Gateway takes a long time to become ready

Cause: on first start it builds a local index cache over every block in the bucket, which is slow on a large bucket.Fix: none needed — wait for it to finish. Raise storeGateway.volumeset.capacity if the cache outgrows the volume.

The Compactor is running but bucket blocks are not being compacted

Cause: after a critical error the Compactor halts — it stays running but stops working, and it is intentionally not restarted for you. Its logs carry the reason:

cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-thanos-compact"} |= "halt"' --limit 50 --since 10m

Fix: resolve the error the log names (see the Compactor documentation), then restart it with cpln workload force-redeployment RELEASE_NAME-thanos-compact --gvc GVC_NAME. If more than one Compactor has ever run against the bucket, stop the extra one first.

Store Gateway or Compactor never starts with storage.type minio

Cause: the dictionary secret named in storage.minio.credentialsSecretName does not exist, so the deployment waits on it with no containers and no logs.Fix: read status.versions[].message with cpln workload get-deployments RELEASE_NAME-thanos-store --gvc GVC_NAME -o yaml, create the named secret as under Prerequisites, and wait for recovery or force a redeployment.

Important Notes

Query has no built-in authentication — keep publicAccess.enabled: false, or front Query with your own authenticating proxy.
Run exactly one Compactor per bucket, across every install and region, and only in a single-location GVC — a second one corrupts the block layout.
Cross-GVC stores need a firewall change on the Prometheus side — see Store Endpoints.
Changing stores takes effect only through cpln helm upgrade.
Disabling the storage tier does not remove existing cloud access from the identity — a previously applied AWS or GCP binding stays until you edit the identity, or uninstall and reinstall the release.
Rotating the MinIO credentials secret is not picked up by running workloads — force a redeployment of RELEASE_NAME-thanos-store and RELEASE_NAME-thanos-compact afterwards.

External References

Thanos Documentation

Official Thanos documentation

Query Component

Global query layer, fan-out, and deduplication reference

Store Gateway Component

Serving historical bucket blocks to the query layer

Compactor Component

Compaction, downsampling, and retention reference

Thanos Template

View the source files, default values, and chart definition

Template Catalog

Templates

Overview

What Gets Created

Prerequisites

Installation

UI

CLI

Terraform

Pulumi

Configuration

Query

Store Endpoints

Access

Long-Term Storage Tier

Object Storage

Connecting

Operations

Upgrading from 1.0.0

Upgrading from 1.1.0

Scaling and Availability

Troubleshooting

Important Notes

External References

Thanos Documentation

Query Component

Store Gateway Component

Compactor Component

Thanos Template

​Overview

​What Gets Created

​Prerequisites

​Installation

UI

CLI

Terraform

Pulumi

​Configuration

​Query

​Store Endpoints

​Access

​Long-Term Storage Tier

​Object Storage

​Connecting

​Operations

​Upgrading from 1.0.0

​Upgrading from 1.1.0

​Scaling and Availability

​Troubleshooting

​Important Notes

​External References

Thanos Documentation

Query Component

Store Gateway Component

Compactor Component

Thanos Template

Overview

What Gets Created

Prerequisites

Installation

Configuration

Query

Store Endpoints

Access

Long-Term Storage Tier

Object Storage

Connecting

Operations

Upgrading from 1.0.0

Upgrading from 1.1.0

Scaling and Availability

Troubleshooting

Important Notes

External References