Prometheus

Overview

Prometheus is the standard open-source metrics database. This template deploys one Prometheus server with the remote-write receiver enabled and its TSDB on a persistent volume: your own senders push metrics in with Prometheus remote_write, or Prometheus scrapes targets you configure, and anything that speaks PromQL (your own Grafana, dashboards, scripts) queries them back. A Thanos sidecar in the same workload, on by default, exposes the Thanos Store API for a Thanos Query tier and can upload TSDB blocks to your own bucket. Use it when you want metrics data you own, or metrics from sources outside your workloads; Control Plane’s built-in metrics keep collecting and dashboarding your workloads either way.

This template does not create a GVC. It deploys into an existing GVC that you already have.

What Gets Created

Resource	Name	Purpose	Optional
Workload	`RELEASE_NAME-prometheus`	Prometheus server (stateful, 1 replica): scrape, remote-write ingest, PromQL and the web UI on port `9095`; container `thanos-sidecar` adds the Store API on `10901` and health on `10902`	Sidecar: `thanos.sidecar.enabled: true` (default)
Volume set	`RELEASE_NAME-prometheus-vs`	TSDB at `/prometheus` (20 GiB, final snapshot on delete, kept 7 days), shared with the sidecar	—
Secret	`RELEASE_NAME-prometheus-config`	The rendered `prometheus.yml`, mounted as a file	—
Secret	`RELEASE_NAME-prometheus-objstore`	Thanos bucket configuration	`thanos.objectStorage.enabled: true`
Identity	`RELEASE_NAME-prometheus-identity`	Workload identity; carries bucket access through your cloud account for AWS and GCP	—
Policy	`RELEASE_NAME-prometheus-policy`	`reveal` on exactly the template’s secrets plus any secrets you name	—

Prometheus listens on 9095 rather than its usual 9090, which is reserved by the platform. There is no public endpoint: Prometheus has no authentication of its own.

Prerequisites

None for a default install. Two optional features need a secret you create first. Secrets are org-level, so no GVC flag is involved.

Optional - remote write with basic auth

For each remoteWrite entry that uses basic auth, create an opaque secret holding the password:

printf '%s' 'YOUR-REMOTE-WRITE-PASSWORD' | cpln secret create-opaque --name REMOTE_WRITE_PASSWORD_SECRET_NAME --encoding plain -f -

Set remoteWrite[].basicAuth.passwordSecretName to this name.

Optional - object storage

Uploading TSDB blocks to a bucket needs the bucket and its access set up first, and an S3-compatible server also needs a credentials secret. The per-provider steps are under Backing Up.

A secret named in values but missing wedges the deployment silently. cpln helm install still succeeds and the workload never becomes ready. cpln logs returns nothing at all, because the container never ran. The only place the reason appears is status.versions[].message, which names the missing secret:

cpln workload get-deployments RELEASE_NAME-prometheus --gvc GVC_NAME -o yaml

Note this is get-deployments; plain cpln workload get has no versions key. Create the missing secret and the deployment recovers on its own, or force a redeployment to skip the wait:

cpln workload force-redeployment RELEASE_NAME-prometheus --gvc GVC_NAME

Installation

A default install needs no values at all:

cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/prometheus \
  --version 1.2.0 --gvc GVC_NAME

Or follow the instructions for your preferred method:

UI

Browse, install, and manage templates visually

CLI

Manage templates from your terminal

Terraform

Declare templates in your Terraform configurations

Pulumi

Declare templates in your Pulumi programs

Configuration

Prometheus Server

image: prom/prometheus:v3.13.1

resources: # Prometheus server; memory scales with active series
  minCpu: 500m
  maxCpu: 1000m
  minMemory: 1Gi
  maxMemory: 2Gi

The block exposes both a reservation and a limit, so the limit is named maxCpu / maxMemory. Raise maxMemory as your number of active series grows.

TSDB and Retention

retention:
  time: 15d # how long samples are kept on the local volume
  size: "" # optional local size cap, e.g. 15GB — empty means no cap
volumeset:
  capacity: 20 # GiB — TSDB volume mounted at /prometheus

With object storage enabled, local retention is only the near-term window; older history lives in your bucket. Keep retention.time at least three times thanos.objectStorage.blockDuration so every block stays on the volume long enough to upload.

Metrics

scrapeInterval: 30s # global scrape interval (self-scrape and extraScrapeConfigs)
externalLabels: {} # identity labels on all uploaded/forwarded series, e.g. region: us-east-1, replica: a
extraScrapeConfigs: "" # raw YAML list of additional scrape_configs entries (see README)

The template always adds the external label prometheus: RELEASE_NAME-prometheus. Your externalLabels keys merge over it, and overriding prometheus is allowed (the HA pair needs it). External labels are attached only to data leaving the server (remote write, the Store API, uploads), never to local query results. To scrape your own workloads, set extraScrapeConfigs to a raw YAML list of scrape_configs entries:

extraScrapeConfigs: |
  - job_name: my-app
    static_configs:
      - targets: ["my-app.my-gvc.cpln.local:8080"]

Use the fully qualified WORKLOAD_NAME.GVC_NAME.cpln.local form for every target inside Control Plane. scrapeInterval, externalLabels, extraScrapeConfigs and remoteWrite are written into a mounted configuration file, and a running replica keeps the file it started with. After a cpln helm upgrade that changes any of them, force a redeployment so the server loads the new configuration:

cpln workload force-redeployment RELEASE_NAME-prometheus --gvc GVC_NAME

Remote Write

remoteWrite: [] # entries: url (required) + optional basicAuth
# - url: http://my-mimir.my-gvc.cpln.local:8080/api/v1/push
#   basicAuth:
#     username: my-user
#     passwordSecretName: my-remote-write-password # opaque secret, must exist BEFORE install

remoteWrite pushes this server’s metrics to any Prometheus-remote-write-compatible store, for example this catalog’s Grafana Mimir template. It is the recommended shape when the Thanos sidecar is off:

thanos:
  sidecar:
    enabled: false
remoteWrite:
  - url: http://my-mimir.my-gvc.cpln.local:8080/api/v1/push
    basicAuth:
      username: my-user
      passwordSecretName: my-remote-write-password # opaque secret, must exist BEFORE install

basicAuth is optional. When set, both username and passwordSecretName are required; the password lives only in the opaque secret from Prerequisites, and the workload identity gets reveal on exactly that secret. A remoteWrite change takes effect only after the forced redeployment described under Metrics.

Thanos Sidecar and Object Storage

thanos:
  sidecar:
    enabled: true # co-located sidecar exposing the Store API (gRPC :10901) for a Thanos Query tier
    image: quay.io/thanos/thanos:v0.42.2
    resources:
      minCpu: 100m
      maxCpu: 250m
      minMemory: 128Mi
      maxMemory: 512Mi
  # Sidecar uploads TSDB blocks to a bucket — recommended for production durability.
  objectStorage:
    enabled: false # requires thanos.sidecar.enabled
    type: aws # options: aws, gcp, minio
    blockDuration: 2h # advanced: TSDB block interval (min=max disables local compaction)

    # AWS S3 — keyless via Control Plane cloud identity.
    aws:
      bucket: my-prometheus-bucket # bucket must already exist
      region: us-east-1
      cloudAccountName: my-s3-cloud-account # Control Plane AWS cloud account
      policyName: my-prometheus-s3-policy # custom IAM policy granting bucket access (bare name)

    # Google Cloud Storage — keyless via Control Plane cloud identity.
    gcp:
      bucket: my-prometheus-bucket # bucket must already exist
      cloudAccountName: my-gcs-cloud-account # Control Plane GCP cloud account

    # Any S3-compatible server (MinIO, R2, Wasabi, …) — static keys + endpoint.
    minio:
      endpoint: my-minio:9000 # host:port, NO scheme — use the FQDN, e.g. my-minio.my-gvc.cpln.local:9000
      insecure: true # true when the endpoint is plain HTTP
      bucket: my-prometheus-bucket # bucket must already exist
      region: us-east-1
      # REQUIRED PREREQUISITE SECRET when type is `minio` — a `dictionary` secret
      # holding exactly `accessKey` and `secretKey`. They are passed to the Thanos
      # sidecar as AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY rather than written
      # into objstore.yml, because a cpln:// reference inside a config file is
      # never resolved.
      credentialsSecretName: my-prometheus-minio-credentials

Turn the sidecar off when you forward metrics with remoteWrite instead. thanos.objectStorage.enabled: true is refused at render unless the sidecar is on, because the sidecar performs the uploads. Set type and fill in only that provider’s block; the setup steps per provider are under Backing Up.

Access

internalAccess:
  type: same-gvc # options: none, same-gvc, same-org, workload-list
  workloads: [] # with workload-list, e.g. //gvc/GVC/workload/NAME — cross-GVC callers allowed

`internalAccess.type`	Description
`none`	No internal access.
`same-gvc`	Allow access from all workloads in the same GVC (default).
`same-org`	Allow access from all workloads in the same org.
`workload-list`	Allow access only from the workloads listed in `internalAccess.workloads`, which may be in other GVCs.

Entries in internalAccess.workloads take the form //gvc/GVC_NAME/workload/WORKLOAD_NAME. Use same-org or workload-list to let a Thanos Query tier in another GVC reach the Store API. Access changes are not instantaneous; re-test after a pause rather than concluding a setting did not work. There is deliberately no public access option. To serve clients outside Control Plane, put your own authenticating proxy in front of Prometheus; the template does not provide one.

Connecting

Everything is internal. Prometheus serves ingest, PromQL and its web UI on port 9095; the Thanos sidecar serves the Store API on 10901.

What	Address	Notes
Remote-write ingest	`http://RELEASE_NAME-prometheus.GVC_NAME.cpln.local:9095/api/v1/write`	Point your senders here
PromQL query and Grafana datasource	`http://RELEASE_NAME-prometheus.GVC_NAME.cpln.local:9095`	For example the Grafana template in the same GVC
Web UI	`http://localhost:9095/query` through a port-forward	Expression browser; also `/targets`, `/config` and `/tsdb-status`
Thanos Store API	`RELEASE_NAME-prometheus.GVC_NAME.cpln.local:10901`	gRPC, `host:port` with no scheme; only with the sidecar on
Credentials	None	Prometheus has no authentication; `internalAccess` decides who can connect

Always use the fully qualified hostname. To open the web UI from your machine, start a port-forward and leave it running:

cpln port-forward RELEASE_NAME-prometheus 9095:9095 --gvc GVC_NAME

Then, in a second terminal, check that the server is ready before you open http://localhost:9095/query in a browser:

curl -s http://localhost:9095/-/ready

The Targets page (/targets) shows every scrape target and the exact error when a scrape fails; it is the first place to look when an expected metric never appears. Configuration (/config) shows the running prometheus.yml, which confirms your extraScrapeConfigs landed. To push metrics here from another Prometheus:

remote_write:
  - url: http://RELEASE_NAME-prometheus.GVC_NAME.cpln.local:9095/api/v1/write

A Thanos Query tier, such as this catalog’s Thanos template, adds RELEASE_NAME-prometheus.GVC_NAME.cpln.local:10901 as a store endpoint. Traffic between locations is billed as egress, so keep the query tier near its stores where practical.

Operations

Backing Up

The volume set keeps a final snapshot for 7 days when the release is uninstalled; there are no scheduled snapshots. For durable history, turn on object storage: the Thanos sidecar uploads each completed TSDB block to your bucket. Set thanos.objectStorage.enabled: true, set thanos.objectStorage.type, then complete the steps for that provider.

AWS S3
Google Cloud Storage
S3-compatible (MinIO, R2, Wasabi)

AWS uses a Control Plane cloud identity, so no keys are stored.

Create a bucket

Create an S3 bucket, for example my-prometheus-bucket. Set thanos.objectStorage.aws.bucket and thanos.objectStorage.aws.region to match.

Create a bucket-scoped IAM policy

In AWS IAM, create a policy, for example my-prometheus-s3-policy, scoped to that bucket:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
      "Resource": [
        "arn:aws:s3:::my-prometheus-bucket",
        "arn:aws:s3:::my-prometheus-bucket/*"
      ]
    }
  ]
}

Set thanos.objectStorage.aws.policyName to the policy name (the bare name, not the ARN).

Create a cloud account

If you do not have one, create a Control Plane cloud account for your AWS account. Set thanos.objectStorage.aws.cloudAccountName to its name.

GCS uses a Control Plane cloud identity, so no keys are stored.

Create a bucket

Create a GCS bucket. Set thanos.objectStorage.gcp.bucket to its name.

Create a cloud account

If you do not have one, create a Control Plane cloud account for your GCP project. Set thanos.objectStorage.gcp.cloudAccountName to its name.

Bucket access

Nothing to author: the template grants the workload identity roles/storage.objectAdmin on exactly that bucket.

An S3-compatible server uses static keys, which you keep in a secret.

Create a bucket and credentials

Create the bucket on your server, and an access key pair that can read and write it. Set thanos.objectStorage.minio.bucket and thanos.objectStorage.minio.region to match.

Create the credentials secret

Create a dictionary secret with exactly the keys accessKey and secretKey:

cpln secret create-dictionary --name MINIO_CREDENTIALS_SECRET_NAME \
  --entry accessKey='YOUR-ACCESS-KEY' \
  --entry secretKey='YOUR-SECRET-KEY'

Set thanos.objectStorage.minio.credentialsSecretName to this name. It must exist before you install or upgrade.

Set the endpoint

Set thanos.objectStorage.minio.endpoint to the S3 API address as host:port with no scheme, for example my-minio.my-gvc.cpln.local:9000. An endpoint containing :// is refused at render. Set thanos.objectStorage.minio.insecure: true when the endpoint is plain HTTP.

Nothing is uploaded until the first block completes, which takes at least one blockDuration after install. Do not change blockDuration unless you know you need to.

Restoring a Backup

No restore procedure has been verified for this template. Blocks in your bucket are not loaded back into this server; they are meant to be read by a Thanos object-storage tier such as the Thanos template’s. Restoring the volume set’s final snapshot into a new install is likewise unverified.

Upgrading from 1.0.0

Version 1.1.0 renamed the resource limits in both blocks: resources.cpu and resources.memory became resources.maxCpu and resources.maxMemory, and the same for thanos.sidecar.resources. The old names are not read any more, so rename them in your values file or the defaults apply. Upgrade in place, passing your values file so nothing resets to its default:

cpln helm upgrade RELEASE_NAME oci://ghcr.io/controlplane-com/templates/prometheus \
  --version 1.2.0 --gvc GVC_NAME \
  -f values.yaml

The values file must use the chart’s nesting, exactly as the blocks in Configuration show. A flat dotted key is accepted without error and silently changes nothing.

Upgrading from 1.1.0

Version 1.2.0 moves the S3-compatible (minio) object-storage keys out of values. thanos.objectStorage.minio.accessKey and thanos.objectStorage.minio.accessSecret are gone; create the dictionary secret from the S3-compatible tab under Backing Up and set thanos.objectStorage.minio.credentialsSecretName to its name. A values file that still carries the old keys is refused at render with:

prometheus: minio.accessKey and minio.accessSecret were REMOVED — they are now a `dictionary` secret you create, named by minio.credentialsSecretName, holding the keys `accessKey` and `secretKey`. Delete them from your values.

AWS and GCP installs are unaffected. Create the secret first, remove the old keys, then run the upgrade command above.

High Availability

Prometheus has no cluster mode, so the template has no replicas setting: one install is one server. The upstream HA pattern is two independent, identically configured servers deduplicated at query time. Install the template twice, for example as releases prom-a and prom-b, with identical values except externalLabels. Values for prom-a:

externalLabels:
  prometheus: my-prom
  replica: a

Values for prom-b:

externalLabels:
  prometheus: my-prom
  replica: b

Senders write to both servers, and a Thanos Query tier deduplicates on the replica label (the default of the Thanos template’s queryReplicaLabels). Both installs must override prometheus to the same value so their label sets differ only in replica.

Troubleshooting

An expected metric never appears

Open the Targets page (/targets) through a port-forward. It lists each scrape target, its last scrape and the exact scrape error. Then check Configuration (/config) to confirm your extraScrapeConfigs entry is in the running config, and that the target uses a fully qualified WORKLOAD_NAME.GVC_NAME.cpln.local address.

Not ready for a long time after a restart

After a restart Prometheus replays its write-ahead log, and /-/ready returns 503 until that finishes. On a large TSDB this can take minutes. It is normal recovery, not a failure; wait rather than redeploying.

External labels do not show in queries

Expected. External labels are attached only to data leaving the server through remote write, the Store API or bucket uploads, never to local query results.

No blocks in the bucket yet

Blocks are cut every thanos.objectStorage.blockDuration (default 2h), so the first upload happens only after the first block completes. If it still has not appeared after that, read the sidecar logs:

cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-prometheus"} |= "ERROR"' --limit 50 --since 10m

A Thanos Query tier in another GVC cannot reach the Store API

Set internalAccess.type to same-org, or to workload-list with the caller’s full link //gvc/GVC_NAME/workload/WORKLOAD_NAME. Use the fully qualified address RELEASE_NAME-prometheus.GVC_NAME.cpln.local:10901 with no scheme.

Upgrade refused with minio.accessKey and minio.accessSecret were REMOVED

Your values still carry the pre-1.2.0 keys. Follow Upgrading from 1.1.0.

Important Notes

Do not open Prometheus to the internet without your own authentication in front. Anyone who can reach port 9095 can read every metric and run any PromQL.
Create every secret you name before installing; a missing one wedges the deployment with nothing in cpln logs.
After an upgrade that changes scrapeInterval, externalLabels, extraScrapeConfigs or remoteWrite, run cpln workload force-redeployment RELEASE_NAME-prometheus --gvc GVC_NAME; the upgrade alone leaves the running server on its old configuration, and there is no hot reload.
With object storage on, keep retention.time at least three times blockDuration; the defaults 15d and 2h satisfy this.
Uninstalling deletes the volume set; only its final snapshot, kept 7 days, and anything already uploaded to your bucket remain.

External References

Prometheus Documentation

Official Prometheus documentation

Prometheus Configuration

Configuration reference, including scrape_configs

Prometheus remote_write

Remote-write tuning and best practices

Thanos Sidecar

Store API and block-upload sidecar reference

Prometheus Template

View the source files, default values, and chart definition

Template Catalog

Templates

Overview

What Gets Created

Prerequisites

Installation

UI

CLI

Terraform

Pulumi

Configuration

Prometheus Server

TSDB and Retention

Metrics

Remote Write

Thanos Sidecar and Object Storage

Access

Connecting

Operations

Backing Up

Restoring a Backup

Upgrading from 1.0.0

Upgrading from 1.1.0

High Availability

Troubleshooting

Important Notes

External References

Prometheus Documentation

Prometheus Configuration

Prometheus remote_write

Thanos Sidecar

Prometheus Template

​Overview

​What Gets Created

​Prerequisites

​Installation

UI

CLI

Terraform

Pulumi

​Configuration

​Prometheus Server

​TSDB and Retention

​Metrics

​Remote Write

​Thanos Sidecar and Object Storage

​Access

​Connecting

​Operations

​Backing Up

​Restoring a Backup

​Upgrading from 1.0.0

​Upgrading from 1.1.0

​High Availability

​Troubleshooting

​Important Notes

​External References

Prometheus Documentation

Prometheus Configuration

Prometheus remote_write

Thanos Sidecar

Prometheus Template

Overview

What Gets Created

Prerequisites

Installation

Configuration

Prometheus Server

TSDB and Retention

Metrics

Remote Write

Thanos Sidecar and Object Storage

Access

Connecting

Operations

Backing Up

Restoring a Backup

Upgrading from 1.0.0

Upgrading from 1.1.0

High Availability

Troubleshooting

Important Notes

External References