OpenTelemetry Collector

Overview

The OpenTelemetry Collector receives, processes and exports telemetry over OTLP. This template runs a stateless collector that normalizes span URLs, derives span metrics, and forwards traces to Control Plane tracing. It can also ingest OTLP metrics and push them to any Prometheus-remote-write store, such as the catalog Prometheus or Mimir templates, and accept authenticated OTLP from senders outside Control Plane (bearer token or mTLS). Control Plane already collects metrics, logs and traces for every workload on its own. Install this collector when you want something the built-in telemetry does not give you: span-derived metrics you query yourself, OTLP metrics landing in a store you own, or an ingestion endpoint for systems that run elsewhere.

This template deploys into an existing GVC that you already have, set with --gvc at install. It never creates or manages a GVC. Traces only reach the collector after you point the GVC tracing provider at it, which restarts every workload in that GVC (see Connecting).

What Gets Created

Resource	Name	Purpose	Optional
Workload	`RELEASE_NAME`	The collector (standard, stateless, `otelCollector.replicas` copies behind one endpoint)	—
Secret	`RELEASE_NAME-conf`	Rendered collector configuration (opaque, mounted as a file)	—
Identity	`RELEASE_NAME`	Workload identity	—
Policy	`RELEASE_NAME`	Grants the identity `reveal` on the config secret and, when auth is on, on your auth secret	—
Direct load balancer	on the `RELEASE_NAME` workload	TCP passthrough on ports `4317` and `4318` for public mTLS ingestion	`publicAccess.enabled: true` with `auth.method: mtls`

Prerequisites

None for a default install. The collector accepts OTLP from inside the GVC without authentication. Public ingestion requires one of two optional auth modes, and each needs a secret that you create before installing:

Bearer auth - create the token secret

An opaque secret holding the token that senders present as Authorization: Bearer <token>:

printf '%s' "$(openssl rand -hex 32)" | cpln secret create-opaque --name TOKEN_SECRET_NAME --encoding plain -f -

Read it back to hand the token to your senders:

cpln secret reveal TOKEN_SECRET_NAME -o yaml

Set auth.method: bearer and auth.bearer.secretName to this name.

mTLS auth - create the certificate secret

A dictionary secret with exactly the keys cert (server certificate), key (server private key) and ca (the CA that signed your client certificates). Paste your PEM files into a file and apply it:

kind: secret
name: MTLS_SECRET_NAME
type: dictionary
data:
  cert: |-
    -----BEGIN CERTIFICATE-----
    ...
  key: |-
    -----BEGIN PRIVATE KEY-----
    ...
  ca: |-
    -----BEGIN CERTIFICATE-----
    ...

cpln apply -f mtls-secret.yaml

Set auth.method: mtls and auth.mtls.secretName to this name.

Create the auth secret before installing, or the deployment wedges silently. The chart refuses to render when auth.method is bearer or mtls and the matching secretName is blank, but a name that points at a secret which does not exist installs and then never starts. The container never runs, so cpln logs returns nothing. The reason appears only in status.versions[].message:

cpln workload get-deployments RELEASE_NAME --gvc GVC_NAME -o yaml

Plain cpln workload get has no versions key. Once the secret exists the deployment recovers on its own after a few minutes, or force it right away:

cpln workload force-redeployment RELEASE_NAME --gvc GVC_NAME

Installation

A default install needs no --set values:

cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/otel-collector \
  --version 1.2.2 --gvc GVC_NAME

Install the template using your preferred method:

UI

Browse, install, and manage templates visually

CLI

Manage templates from your terminal

Terraform

Declare templates in your Terraform configurations

Pulumi

Declare templates in your Pulumi programs

Configuration

Collector

otelCollector:
  image: otel/opentelemetry-collector-contrib:0.157.0
  mode: simple # simple: structured config from the knobs below | advanced: advanced.config is used verbatim
  replicas: 1 # stateless; 2+ runs an HA ingestion pool behind the same endpoint
  resources:
    cpu: 200m
    memory: 256Mi

resources sets the container limit only. replicas is a fixed count: the collector does not autoscale.

Simple Mode

otelCollector:
  simple:
    processors:
      transform:
        traceStatements: # Normalizes dynamic span URLs to a fixed value. Substitute PLACEHOLDER, e.g. 'user/profile'
          - replace_pattern(span.attributes["http.url"], "^.*(PLACEHOLDER).*$", "/PLACEHOLDER")
    spanmetrics:
      histogram: # Span-duration histogram buckets — tune to your SLO thresholds
        # Buckets MUST be duration strings ("250ms", "1s"). Bare numbers are
        # parsed as NANOSECONDS by the collector, silently breaking the histogram.
        buckets: ["1ms", "5ms", "10ms", "25ms", "50ms", "100ms", "250ms", "500ms", "1s", "2.5s", "5s"]
        unit: ms # ms or s (output unit of the recorded histogram)

Add one replace_pattern statement per URL shape you want to collapse, so span cardinality stays manageable:

otelCollector:
  simple:
    processors:
      transform:
        traceStatements:
          - replace_pattern(span.attributes["http.url"], "^.*(/user/profile).*$", "/user/profile")
          - replace_pattern(span.attributes["http.url"], "^.*(/user/settings/.*).*$", "/user/settings/.*")

Span metrics are grouped by http.url, http.method and http.status_code. With unit: ms the histogram is named traces_span_metrics_duration_milliseconds_bucket, and with unit: s it is traces_span_metrics_duration_seconds_bucket.

Advanced Mode

With mode: advanced, the collector runs advanced.config exactly as written and ignores the simple-mode knobs. The shipped config is a working starting point equivalent to simple mode:

otelCollector:
  advanced:
    config: |
      extensions:
        health_check:
          endpoint: 0.0.0.0:13133
        pprof:
          endpoint: 0.0.0.0:8180

      receivers:
        otlp:
          protocols:
            grpc:
              endpoint: 0.0.0.0:4317
            http:
              endpoint: 0.0.0.0:4318

      processors:
        batch:
        resource:
          attributes:
            - key: workload
              from_attribute: service.name
              action: insert
        transform:
          trace_statements:
            - context: span
              statements:
                - replace_pattern(span.attributes["http.url"], "^.*(PLACEHOLDER).*$", "/PLACEHOLDER")

      connectors:
        spanmetrics:
          dimensions:
            - name: http.url
            - name: http.method
            - name: http.status_code
          histogram:
            explicit:
              buckets: ["1ms", "5ms", "10ms", "25ms", "50ms", "100ms", "250ms", "500ms", "1s", "2.5s", "5s"]
            unit: ms

      exporters:
        otlp:
          endpoint: http://tracing.controlplane:80
          tls:
            insecure: true
        prometheus:
          endpoint: 0.0.0.0:8889

      service:
        pipelines:
          traces:
            receivers: [otlp]
            processors: [resource, transform, batch]
            exporters: [otlp, spanmetrics]
          metrics:
            receivers: [spanmetrics]
            processors: [batch]
            exporters: [prometheus]
        extensions: [pprof, health_check]
        telemetry:
          logs:
            level: INFO

Rules for an advanced config:

Keep health_check on 0.0.0.0:13133. The chart only adds readiness and liveness probes when the health check listens on 0.0.0.0.
Container ports come from receivers.otlp. The chart reads the gRPC and HTTP endpoints of the otlp receiver and declares those ports.
For public ingestion, bind your authenticated receiver to 0.0.0.0:4318 (HTTP) and 0.0.0.0:4319 (gRPC). The auth.* and publicAccess.* knobs still mount the secret, grant reveal, and set the firewall and load balancer. Token and certificate files are mounted at /etc/otel-collector/auth/token and /etc/otel-collector/tls/server.crt, server.key and ca.crt.
metrics.enabled: true is refused in advanced mode. Add the remote-write exporter and pipeline to your config instead:

exporters:
  prometheus_remote_write:
    endpoint: http://RELEASE_NAME-prometheus.GVC_NAME.cpln.local:9095/api/v1/write
service:
  pipelines:
    metrics/otlp:
      receivers: [otlp]
      processors: [resource, batch]
      exporters: [prometheus_remote_write]

Metrics Ingestion

metrics:
  enabled: false # adds an OTLP→remote-write metrics pipeline (simple mode; in advanced mode define it in advanced.config)
  remoteWrite:
    endpoint: http://YOUR_WORKLOAD.YOUR_GVC.cpln.local:9095/api/v1/write # any Prometheus-remote-write-compatible URL

With metrics.enabled: true, OTLP metrics sent to the collector are pushed to metrics.remoteWrite.endpoint. One collector pushes to one store, so install the template again for each additional target. Remote-write endpoints of the sibling templates in the same GVC:

Target	`metrics.remoteWrite.endpoint`
Prometheus	`http://RELEASE_NAME-prometheus.GVC_NAME.cpln.local:9095/api/v1/write`
Mimir	`http://RELEASE_NAME-mimir.GVC_NAME.cpln.local:8080/api/v1/push`

Here RELEASE_NAME is the release name of the Prometheus or Mimir install, not of this collector. A Thanos query tier reads from Prometheus, so point the collector at Prometheus and query through Thanos. Changing metrics.enabled or metrics.remoteWrite updates only the collector’s configuration secret, which a running replica does not re-read. After the cpln helm upgrade, run cpln workload force-redeployment RELEASE_NAME --gvc GVC_NAME.

Substitute both the workload name and the GVC in the endpoint. The install is rejected while YOUR_WORKLOAD, YOUR_GVC or my-gvc.cpln.local remains in metrics.remoteWrite.endpoint (simple mode with metrics on) or anywhere in otelCollector.advanced.config (advanced mode). An endpoint that does not resolve would otherwise drop every metric while the collector still answers senders with success.

Ingestion Auth

auth:
  method: none # none | bearer | mtls — must not be none when publicAccess is enabled
  bearer:
    secretName: "" # opaque secret holding the token (e.g. my-otel-ingest-token) — create BEFORE install
  mtls:
    secretName: "" # dictionary secret with keys cert, key, ca (server cert/key + client CA) — create BEFORE install

When method is bearer or mtls, a separate authenticated receiver listens on 4318 (HTTP) and 4319 (gRPC). The plain gRPC receiver on 4317 stays unauthenticated for the GVC tracing provider and is never exposed publicly.

bearer — senders send Authorization: Bearer <token>. Public bearer ingestion uses the canonical HTTPS endpoint and is OTLP/HTTP only; use mTLS for public gRPC.
mtls — the collector presents your server certificate and requires a client certificate signed by your CA. Public mTLS ingestion uses the direct load balancer, and the canonical HTTPS endpoint stops accepting traffic in this mode.

Access

publicAccess:
  enabled: false # public ingestion: bearer → canonical https endpoint; mtls → direct TCP 4317/4318
  allowedCidrs: [] # REQUIRED when enabled — sender CIDRs, e.g. ["203.0.113.0/24"]; use ["0.0.0.0/0"] to explicitly allow all
internalAccess:
  type: same-gvc # who inside the org can reach the collector internally: none | same-gvc | same-org

The install is refused when publicAccess.enabled is true with auth.method: none, or with an empty allowedCidrs. Opening to every address takes an explicit ["0.0.0.0/0"]. A public bearer install looks like this:

cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/otel-collector \
  --version 1.2.2 --gvc GVC_NAME \
  --set auth.method=bearer \
  --set auth.bearer.secretName=TOKEN_SECRET_NAME \
  --set publicAccess.enabled=true \
  --set 'publicAccess.allowedCidrs={203.0.113.0/24}'

Connecting

What	Address	Notes
OTLP gRPC, in the GVC	`RELEASE_NAME.GVC_NAME.cpln.local:4317`	No auth. The GVC tracing provider target
OTLP HTTP, in the GVC	`http://RELEASE_NAME.GVC_NAME.cpln.local:4318`	No auth by default. Bearer token when `auth.method: bearer`. In mTLS mode use gRPC `4317` from inside the GVC
OTLP HTTP, public (bearer)	`https://CANONICAL_ENDPOINT/v1/traces` and `/v1/metrics`	`Authorization: Bearer <token>`
OTLP, public (mTLS)	direct load balancer address, port `4318` (HTTP) and `4317` (gRPC)	Client certificate signed by your CA
Span metrics scrape	`http://RELEASE_NAME.GVC_NAME.cpln.local:8889/metrics`	Prometheus format, no auth

Read the canonical endpoint from status.canonicalEndpoint:

cpln workload get RELEASE_NAME --gvc GVC_NAME -o yaml

The bearer token is the value in your token secret (cpln secret reveal TOKEN_SECRET_NAME -o yaml).

Send Traces From the GVC

Point the GVC OpenTelemetry tracing provider at the collector gRPC port. In the console, open the GVC, enable tracing with the OpenTelemetry provider, and set the endpoint. The resulting GVC spec is:

spec:
  tracing:
    provider:
      otel:
        endpoint: RELEASE_NAME.GVC_NAME.cpln.local:4317

Saving it restarts every workload in the GVC. Do not cpln apply a GVC file that holds only this block: an apply replaces the whole GVC spec, including its locations. See OpenTelemetry Tracing Provider for the field reference. Applications instrumented with an OpenTelemetry SDK can also export OTLP straight to the in-GVC addresses above.

Verify the Receiver

Forward the OTLP HTTP port and post an empty trace batch. An HTTP 200 means the receiver accepts OTLP. This check applies to a default install (auth.method: none) and was not exercised against a live install for this page:

cpln port-forward RELEASE_NAME 4318:4318 --gvc GVC_NAME

curl -s -o /dev/null -w '%{http_code}\n' -X POST http://127.0.0.1:4318/v1/traces \
  -H 'Content-Type: application/json' -d '{}'

Operations

Upgrading from 1.0.x

The default otelCollector.mode is simple since 1.1.0 (it was advanced). If you customized advanced.config and relied on the old default, set otelCollector.mode: advanced explicitly.
Histogram buckets must be duration strings. Convert numeric buckets carried over from 1.0.x, for example 250 to "250ms".

cpln helm upgrade RELEASE_NAME oci://ghcr.io/controlplane-com/templates/otel-collector \
  --version 1.2.2 --gvc GVC_NAME \
  --set otelCollector.mode=advanced

Pass the same --set values or values file you installed with, so the upgrade does not reset other settings to their defaults.

Scaling and Availability

The collector keeps no state, so replicas need no coordination. Set otelCollector.replicas to 2 or more to keep ingestion running through rolling restarts and the loss of a replica:

cpln helm upgrade RELEASE_NAME oci://ghcr.io/controlplane-com/templates/otel-collector \
  --version 1.2.2 --gvc GVC_NAME \
  --set otelCollector.replicas=2

Rotating the Bearer Token

Apply the secret with a new value, then force a redeployment. A running replica keeps the old token until it restarts:

kind: secret
name: TOKEN_SECRET_NAME
type: opaque
data:
  encoding: plain
  payload: NEW-VALUE

cpln apply -f rotate.yaml
cpln workload force-redeployment RELEASE_NAME --gvc GVC_NAME

Troubleshooting

Metrics never arrive in Prometheus or Mimir

Cause: the remote-write endpoint does not resolve or is unreachable. Senders still get a success response because the collector accepts the data before exporting it.Fix: search the collector logs for the exporter failure, then correct metrics.remoteWrite.endpoint to the fully qualified address of the store:

cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME"} |= "Dropping data"' --limit 50 --since 10m

A metric name is missing but the endpoint is correct

Cause: remote write adds unit and type suffixes during the OTLP to Prometheus translation. A metric sent as my_metric is stored as my_metric_seconds or my_metric_ratio depending on its OTLP unit, and a counter sent as demo_counter arrives as demo_counter_total.Fix: query the suffixed name in Prometheus or Grafana.

Traces stop after enabling auth

Cause: the GVC tracing provider was pointed at port 4318, which belongs to the authenticated receiver once auth is on.Fix: set the GVC tracing endpoint to RELEASE_NAME.GVC_NAME.cpln.local:4317, the plain gRPC receiver.

Install fails with publicAccess.enabled true with auth.method none

Error: otel-collector: you set publicAccess.enabled: true with auth.method: none.Fix: set auth.method to bearer or mtls with the matching secretName, or keep publicAccess.enabled: false and send from inside the GVC.

Install fails because allowedCidrs is empty

Error: otel-collector: publicAccess.allowedCidrs must be non-empty when publicAccess.enabledFix: list your sender CIDRs in publicAccess.allowedCidrs, or ["0.0.0.0/0"] to allow every address.

Install fails because the endpoint still contains a placeholder

Error: otel-collector: metrics.remoteWrite.endpoint still contains the placeholder (simple mode) or otel-collector: otelCollector.advanced.config still contains the placeholder (advanced mode).Fix: replace both the workload name and the GVC in the exporter endpoint.

In-GVC senders fail the TLS handshake in mTLS mode

Cause: ports 4318 and 4319 serve mTLS for external clients through the direct load balancer.Fix: send from inside the GVC to the plain gRPC receiver on RELEASE_NAME.GVC_NAME.cpln.local:4317.

Important Notes

Point the GVC tracing provider at RELEASE_NAME.GVC_NAME.cpln.local:4317 after install; saving it restarts every workload in the GVC.
Histogram buckets must be duration strings ("250ms", "1s"); bare numbers are read as nanoseconds and break the histogram.
Create the auth secret before installing; a missing secret wedges the deployment with no logs.
Public ingestion always needs auth.method set and a non-empty allowedCidrs.
In mTLS mode the canonical HTTPS endpoint stops accepting traffic; public clients use the direct load balancer.
One collector pushes to one remote-write store; install again for each extra target.
Rotating a token or certificate secret takes effect only after cpln workload force-redeployment.

Template Catalog

Templates

Overview

What Gets Created

Prerequisites

Installation

UI

CLI

Terraform

Pulumi

Configuration

Collector

Simple Mode

Advanced Mode

Metrics Ingestion

Ingestion Auth

Access

Connecting

Send Traces From the GVC

Verify the Receiver

Operations

Upgrading from 1.0.x

Scaling and Availability

Rotating the Bearer Token

Troubleshooting

Important Notes

External References

​Overview

​What Gets Created

​Prerequisites

​Installation

UI

CLI

Terraform

Pulumi

​Configuration

​Collector

​Simple Mode

​Advanced Mode

​Metrics Ingestion

​Ingestion Auth

​Access

​Connecting

​Send Traces From the GVC

​Verify the Receiver

​Operations

​Upgrading from 1.0.x

​Scaling and Availability

​Rotating the Bearer Token

​Troubleshooting

​Important Notes

​External References

Overview

What Gets Created

Prerequisites

Installation

Configuration

Collector

Simple Mode

Advanced Mode

Metrics Ingestion

Ingestion Auth

Access

Connecting

Send Traces From the GVC

Verify the Receiver

Operations

Upgrading from 1.0.x

Scaling and Availability

Rotating the Bearer Token

Troubleshooting

Important Notes

External References