OpenTelemetry Collector

Overview

The OpenTelemetry Collector receives, processes, and exports telemetry over OTLP. This template deploys a stateless collector that feeds Control Plane’s native tracing, and can additionally ingest OTLP metrics and push them to any Prometheus-remote-write-compatible store — such as this catalog’s Prometheus template — with optional authenticated public ingestion (bearer token or mTLS).

Architecture

Collector — A standard, stateless workload. replicas copies run behind one endpoint; 2 or more forms an HA ingestion pool that keeps accepting telemetry through rolling restarts and replica loss.
Traces — Applications send OTLP to the collector; it normalizes spans, derives span metrics, and forwards traces to the Control Plane tracing backend.
Metrics (optional) — An OTLP-to-prometheus_remote_write pipeline pushes ingested metrics to the store you configure.
Public ingestion (optional) — Bearer-token auth over the canonical HTTPS endpoint, or mutual TLS over a direct TCP load balancer. Public access always requires auth plus an explicit CIDR allowlist.

What Gets Created

Standard Collector Workload — Runs the collector in either a simple (knob-driven) or advanced (full custom config) mode.
Secret — An opaque secret holding the rendered collector configuration, mounted as a file.
Identity & Policy — An identity bound to the workload with reveal access to the config secret and — only when auth is enabled — the auth secret you name.
Direct Load Balancer (optional) — TCP passthrough on ports 4317/4318, created only for public mTLS ingestion.

The workload, secret, identity and policy each carry a description explaining its role in the console (the direct load balancer is a field on the workload, not a resource of its own). The workload’s description follows the mode: in simple mode it names the ports the collector binds, and in advanced mode it says the configuration is user-supplied and does not assert the ingestion ports, since those are then yours to choose.

This template does not create a GVC. You will need to deploy it into an existing GVC and enable tracing at the GVC level after installing, specifying the target workload and port. This will trigger a restart of all workloads in the GVC.

Prerequisites

None for a default install. The two auth modes have prerequisites:

Bearer auth (auth.method: bearer) — an opaque secret holding the token, created before install (a missing secret leaves the deployment waiting on it). Set auth.bearer.secretName to its name:
openssl rand -hex 32 | tr -d '\n' | cpln secret create-opaque --name my-otel-ingest-token --encoding plain -f -

mTLS auth (auth.method: mtls) — a dictionary secret created before install with exactly the keys cert (server certificate), key (server private key), and ca (the CA that signed your client certificates). Set auth.mtls.secretName to its name:

kind: secret
name: my-otel-mtls-certs
type: dictionary
data:
  cert: |-
    -----BEGIN CERTIFICATE----- ...
  key: |-
    -----BEGIN PRIVATE KEY----- ...
  ca: |-
    -----BEGIN CERTIFICATE----- ...

Install the template using your preferred method:

UI

Browse, install, and manage templates visually

CLI

Manage templates from your terminal

Terraform

Declare templates in your Terraform configurations

Pulumi

Declare templates in your Pulumi programs

Configuration

The default otelCollector.mode changed from advanced to simple in template version 1.1.0. If you customized advanced.config while relying on the old default, set mode: advanced explicitly when upgrading.

The default values.yaml for this template:

# IMPORTANT:
# - Traces: after installing, enable tracing at the GVC level (triggers a restart of all workloads in the GVC).
# - Metrics: set metrics.enabled=true and point remoteWrite.endpoint at any Prometheus-remote-write-compatible store.
otelCollector:
  image: otel/opentelemetry-collector-contrib:0.157.0
  mode: simple # simple: structured config from the knobs below | advanced: advanced.config is used verbatim
  replicas: 1 # stateless; 2+ runs an HA ingestion pool behind the same endpoint
  resources:
    cpu: 200m
    memory: 256Mi

  # Simple Mode
  simple:
    processors:
      transform:
        traceStatements: # Normalizes dynamic span URLs to a fixed value. Substitute PLACEHOLDER, e.g. 'user/profile'
          - replace_pattern(span.attributes["http.url"], "^.*(PLACEHOLDER).*$", "/PLACEHOLDER")
    spanmetrics:
      histogram: # Span-duration histogram buckets — tune to your SLO thresholds
        # Buckets MUST be duration strings ("250ms", "1s"). Bare numbers are
        # parsed as NANOSECONDS by the collector, silently breaking the histogram.
        buckets: ["1ms", "5ms", "10ms", "25ms", "50ms", "100ms", "250ms", "500ms", "1s", "2.5s", "5s"]
        unit: ms # ms or s (output unit of the recorded histogram)

  # Advanced Mode — full collector config, used verbatim when mode: advanced.
  # Public ingestion contract in advanced mode: bind your authed receiver to
  # 0.0.0.0:4318 (HTTP) / 0.0.0.0:4319 (gRPC); keep health_check on 0.0.0.0:13133 for probes.
  advanced:
    config: |
      extensions:
        health_check:
          endpoint: 0.0.0.0:13133
        pprof:
          endpoint: 0.0.0.0:8180

      receivers:
        otlp:
          protocols:
            grpc:
              endpoint: 0.0.0.0:4317
            http:
              endpoint: 0.0.0.0:4318

      processors:
        batch:
        resource:
          attributes:
            - key: workload
              from_attribute: service.name
              action: insert
        transform:
          trace_statements:
            - context: span
              statements:
                - replace_pattern(span.attributes["http.url"], "^.*(PLACEHOLDER).*$", "/PLACEHOLDER")

      connectors:
        spanmetrics:
          dimensions:
            - name: http.url
            - name: http.method
            - name: http.status_code
          histogram:
            explicit:
              buckets: ["1ms", "5ms", "10ms", "25ms", "50ms", "100ms", "250ms", "500ms", "1s", "2.5s", "5s"]
            unit: ms

      exporters:
        otlp:
          endpoint: http://tracing.controlplane:80
          tls:
            insecure: true
        prometheus:
          endpoint: 0.0.0.0:8889

      service:
        pipelines:
          traces:
            receivers: [otlp]
            processors: [resource, transform, batch]
            exporters: [otlp, spanmetrics]
          metrics:
            receivers: [spanmetrics]
            processors: [batch]
            exporters: [prometheus]
        extensions: [pprof, health_check]
        telemetry:
          logs:
            level: INFO

# ─── Metrics Ingestion (simple mode) ─────────────────────────────────────────
metrics:
  enabled: false # adds an OTLP→remote-write metrics pipeline (simple mode; in advanced mode define it in advanced.config)
  remoteWrite:
    endpoint: http://YOUR_WORKLOAD.YOUR_GVC.cpln.local:9095/api/v1/write # any Prometheus-remote-write-compatible URL

# ─── Ingestion Auth ──────────────────────────────────────────────────────────
auth:
  method: none # none | bearer | mtls — must not be none when publicAccess is enabled
  bearer:
    secretName: "" # opaque secret holding the token (e.g. my-otel-ingest-token) — create BEFORE install
  mtls:
    secretName: "" # dictionary secret with keys cert, key, ca (server cert/key + client CA) — create BEFORE install

# ─── Access ──────────────────────────────────────────────────────────────────
publicAccess:
  enabled: false # public ingestion: bearer → canonical https endpoint; mtls → direct TCP 4317/4318
  allowedCidrs: [] # REQUIRED when enabled — sender CIDRs, e.g. ["203.0.113.0/24"]; use ["0.0.0.0/0"] to explicitly allow all
internalAccess:
  type: same-gvc # who inside the org can reach the collector internally: none | same-gvc | same-org

Mode and Replicas

otelCollector.mode — Set to simple (the default) for a structured configuration generated from the knobs below, or advanced to provide a fully customized configuration. In advanced mode, the entire otelCollector.advanced.config is passed to the collector verbatim.
otelCollector.replicas — The collector is stateless, so replicas need no coordination. Set 2 or more for an HA ingestion pool behind the same endpoint; ingestion keeps succeeding through rolling restarts and replica loss.

Simple Mode

simple.processors.transform.traceStatements — A list of replace_pattern statements that normalize dynamic URL segments in http.url span attributes. Replace PLACEHOLDER with the path segment you want to collapse, e.g. user/profile.
simple.spanmetrics.histogram.buckets — Span-duration histogram bucket boundaries. Tune these to align with your SLO thresholds.
simple.spanmetrics.histogram.unit — Output unit of the recorded histogram. Choose ms (milliseconds) or s (seconds).

Histogram buckets must be duration strings ("250ms", "1s"). Bare numbers are parsed as nanoseconds by the collector, silently collapsing every span into the +Inf bucket. If you carried numeric buckets over from a 1.0.x install, convert them (e.g. 250 → "250ms") when upgrading.

Metrics Ingestion

metrics:
  enabled: true   # adds an OTLP → prometheus_remote_write pipeline (simple mode only)
  remoteWrite:
    endpoint: http://YOUR_WORKLOAD.YOUR_GVC.cpln.local:9095/api/v1/write  # any Prometheus-remote-write-compatible URL

With metrics.enabled: true, OTLP metrics posted to the collector (internally, or publicly with auth) are pushed to the store at metrics.remoteWrite.endpoint. Any Prometheus-remote-write-compatible URL works — for example, this catalog’s Prometheus template receives remote write at http://RELEASE_NAME-prometheus.GVC_NAME.cpln.local:9095/api/v1/write, and a Thanos query tier on top gives you a global view of what lands there. One collector pushes to one store; run multiple installs for multiple targets.

Substitute both halves of the endpoint — the workload name and the GVC. Since template version 1.2.2 a leftover YOUR_WORKLOAD, YOUR_GVC or my-gvc.cpln.local is rejected at install, in metrics.remoteWrite.endpoint when metrics.enabled: true (simple mode) or anywhere in otelCollector.advanced.config (advanced mode); the message names the placeholder it found and nothing is created, because the check runs before any resource is applied. A default install is unaffected — metrics.enabled is false, so the shipped placeholder endpoint installs as-is.The check exists because the alternative is worse. An endpoint that does not resolve fails silently at the last hop: the sender receives HTTP 200, the collector accepts and batches the data, and then the exporter logs Exporting failed. Dropping data. with a dropped_items count and discards it. The workload stays ready, no status surface reports a problem, and Grafana shows an empty metric picker with no error. If metrics never arrive and nothing looks wrong, check the exporter endpoint first:

cpln logs '{gvc="GVC_NAME", workload="WORKLOAD_NAME"}' | grep -i 'dropping data'

A GVC genuinely named my-gvc also trips the check — a deliberate trade of one loud, legible rejection against a failure mode that drops every metric in silence.

Remote write also renames your metrics. The exporter appends a unit and type suffix following the OTLP-to-Prometheus translation rules, so a metric your application emits as my_metric is stored as my_metric_ratio, my_metric_seconds and so on according to its OTLP unit, and a counter sent as demo_counter arrives as demo_counter_total. Query the suffixed name in Prometheus, not the one your app emits. This is the second thing that hides metrics after an unsubstituted endpoint, and the symptom is identical — you search the Grafana metric picker for the name your app sends and find nothing — so check it next when the endpoint turns out to be fine.

Ingestion Auth

auth:
  method: bearer  # none | bearer | mtls — required (not none) for public ingestion
  bearer:
    secretName: my-otel-ingest-token  # opaque secret created BEFORE install (see Prerequisites)
  mtls:
    secretName: my-otel-mtls-certs    # dictionary secret (cert/key/ca) created BEFORE install

Auth applies to a dedicated otlp/ingest receiver on 4318 (HTTP) / 4319 (gRPC). The plain gRPC :4317 receiver stays unauthenticated for the GVC tracing integration and is never exposed publicly.

bearer — Senders present Authorization: Bearer <token>; requests without the token (or with the wrong one) are rejected with 401. Public bearer ingestion runs over the canonical HTTPS endpoint and is HTTP-only — OTLP/gRPC does not pass the canonical endpoint; use mTLS for public gRPC.
mtls — The collector serves your certificate and requires a client certificate signed by your CA; the TLS handshake itself rejects unauthenticated clients. Public mTLS ingestion runs over a direct TCP load balancer on 4318 (HTTP) and 4317 (gRPC); the canonical https:// endpoint intentionally stops accepting traffic in this mode.

Access

publicAccess:
  enabled: true                     # bearer → canonical https endpoint; mtls → direct TCP 4317/4318
  allowedCidrs: ["203.0.113.0/24"]  # REQUIRED when enabled; use ["0.0.0.0/0"] to explicitly allow all
internalAccess:
  type: same-gvc                    # none | same-gvc | same-org

Public ingestion always requires auth: enabling publicAccess with auth.method: none or an empty allowedCidrs fails at install — opening to the world requires an explicit ["0.0.0.0/0"]. internalAccess.type controls which workloads inside the org can reach the collector internally (none, same-gvc, or same-org).

Advanced Mode

The advanced.config block accepts a full OpenTelemetry Collector configuration and is fully authoritative. The shipped default config provides:

Component	Description
`receivers.otlp`	Accepts telemetry from your applications over gRPC on port `4317` and HTTP on `4318`
`processors.resource`	Copies `service.name` into a `workload` attribute for easier filtering
`processors.transform`	Normalizes dynamic URL segments in span attributes
`connectors.spanmetrics`	Derives duration histograms and request metrics from trace spans
`exporters.otlp`	Forwards traces to the Control Plane tracing backend
`exporters.prometheus`	Exposes span-derived metrics for scraping on port `8889`

The metrics.* knobs are refused in advanced mode (the install fails rather than silently ignoring them). To ingest metrics, add the pipeline to your config yourself — substituting both the workload name and the GVC in the exporter endpoint, which is checked at install (see Metrics Ingestion):

exporters:
  prometheus_remote_write:
    endpoint: http://YOUR_WORKLOAD.YOUR_GVC.cpln.local:9095/api/v1/write
service:
  pipelines:
    metrics/otlp:
      receivers: [otlp]
      processors: [resource, batch]
      exporters: [prometheus_remote_write]

The auth.* and publicAccess.* knobs still wire the secret mounts, reveal policy, firewall, and load balancer in advanced mode, but your config must bind the authed receiver to 0.0.0.0:4318 (HTTP) / 0.0.0.0:4319 (gRPC) — certificate and token files are mounted at /etc/otel-collector/tls/{server.crt,server.key,ca.crt} and /etc/otel-collector/auth/token. Keep health_check on 0.0.0.0:13133 or readiness probes are skipped.

Setting Up Tracing

1. Normalize URL patterns (recommended) If your application has dynamic URL segments, use the transform processor to normalize them into fixed patterns. This keeps span cardinality manageable and makes your traces queryable. In either Simple or Advanced config, replace each occurrence of PLACEHOLDER with each path segment you want to collapse:

transform:
  trace_statements:
    - context: span
      statements:
        # Static path
        - replace_pattern(span.attributes["http.url"], "^.*(/user/profile).*$", "/user/profile")
        # Wildcard path
        - replace_pattern(span.attributes["http.url"], "^.*(/user/settings/.*).*$", "/user/settings/.*")

Add one statement per pattern you want to normalize. 2. Install the OpenTelemetry SDK in your application Your application must emit traces using the OpenTelemetry SDK. Install the SDK and configure it to export via OTLP/gRPC to:

http://<otel-collector-workload-name>.<gvc-name>.cpln.local:4317

3. Enable tracing at the GVC level Navigate to the GVC where the OpenTelemetry Collector is installed and enable tracing. Set the target workload on port 4317. This triggers a restart of all workloads in the GVC so they begin emitting traces to the collector. 4. Query span metrics in Grafana Span metrics are exposed via Prometheus on port 8889 of the collector workload. Use the spanmetrics connector’s output to build dashboards and alerts. The metric name suffix automatically changes based on the unit defined in your configuration:

traces_span_metrics_duration_milliseconds_bucket{http_url="/your-endpoint"}

If you changed the unit to s in the histogram, the metric name changes to:

traces_span_metrics_duration_seconds_bucket{http_url="/your-endpoint"}

Span-metric series first appear roughly 60–90 seconds after the first span — the spanmetrics connector flushes on a 60-second interval, and :8889 serves an empty response until then.

Connecting

Endpoint	Address	Auth
In-GVC OTLP gRPC (traces + metrics)	`RELEASE_NAME.GVC_NAME.cpln.local:4317`	None (GVC tracing target)
In-GVC OTLP HTTP	`http://RELEASE_NAME.GVC_NAME.cpln.local:4318`	None, or bearer when auth is on (in mTLS mode use plain gRPC `:4317` instead)
Public OTLP HTTP (bearer)	`https://CANONICAL_ENDPOINT/v1/traces`, `/v1/metrics`	`Authorization: Bearer <token>`
Public OTLP (mTLS)	`DIRECT_LB_ENDPOINT:4318` (HTTP), `:4317` (gRPC)	Client certificate signed by your CA
Spanmetrics scrape	`http://RELEASE_NAME.GVC_NAME.cpln.local:8889/metrics`	None

The canonical endpoint (and the direct load balancer endpoint in mTLS mode) is in status.canonicalEndpoint of cpln workload get RELEASE_NAME -o yaml. The bearer token is whatever you stored in your prerequisite secret. Use the full .cpln.local hostname for in-GVC senders.

Important Notes

The default mode changed from advanced to simple in template version 1.1.0 — if you customized advanced.config while relying on the old default, set mode: advanced explicitly when upgrading.
Histogram buckets must be duration strings ("250ms", "1s") — bare numbers are parsed as nanoseconds, which silently breaks the histogram.
Enable tracing at the GVC level after install (target the collector workload, port 4317); this restarts all workloads in the GVC.
Public ingestion requires auth: publicAccess.enabled with auth.method: none or an empty allowedCidrs fails at install — opening to the world requires an explicit ["0.0.0.0/0"].
Create the auth secret before installing — a missing secret leaves the deployment waiting on it.
mTLS uses the direct load balancer (raw TCP), not the canonical endpoint; in mTLS mode the canonical https:// endpoint intentionally stops accepting traffic.
In mTLS mode, in-GVC senders must use the plain internal gRPC port :4317 — the TLS-terminating ingest ports (4318/4319) are reachable only by external clients via the direct load balancer.
Substitute both the workload name and the GVC in the remote-write endpoint. A leftover placeholder is rejected at install since version 1.2.2, but the reason matters: an unresolvable endpoint drops metrics silently — the sender gets 200, the collector logs Exporting failed. Dropping data., and nothing else reports a problem. Diagnose with cpln logs '{gvc="GVC_NAME", workload="WORKLOAD_NAME"}' | grep -i 'dropping data'.
Remote write appends a unit and type suffix to metric names. A metric sent as my_metric is stored as my_metric_ratio, my_metric_seconds and so on following its OTLP unit, and a counter sent as demo_counter becomes demo_counter_total — query the suffixed name in Prometheus, not the one your app emits. Like an unsubstituted endpoint, this shows up as an empty Grafana metric picker.
One collector pushes to one remote-write store; run multiple installs for multiple targets.

External References

OTel Collector Documentation

Official OpenTelemetry Collector documentation

OTel SDK Setup

Language-specific SDK guides for instrumenting your application

Spanmetrics Connector

Reference for the spanmetrics connector configuration

Prometheus Remote Write Exporter

Reference for the prometheusremotewrite exporter

OTLP Specification

The OpenTelemetry protocol specification

OTel Collector Template

View the source files, default values, and chart definition

Template Catalog

Templates

Overview

Architecture

What Gets Created

Prerequisites

UI

CLI

Terraform

Pulumi

Configuration

Mode and Replicas

Simple Mode

Metrics Ingestion

Ingestion Auth

Access

Advanced Mode

Setting Up Tracing

Connecting

Important Notes

External References

OTel Collector Documentation

OTel SDK Setup

Spanmetrics Connector

Prometheus Remote Write Exporter

OTLP Specification

OTel Collector Template

​Overview

​Architecture

​What Gets Created

​Prerequisites

UI

CLI

Terraform

Pulumi

​Configuration

​Mode and Replicas

​Simple Mode

​Metrics Ingestion

​Ingestion Auth

​Access

​Advanced Mode

​Setting Up Tracing

​Connecting

​Important Notes

​External References

OTel Collector Documentation

OTel SDK Setup

Spanmetrics Connector

Prometheus Remote Write Exporter

OTLP Specification

OTel Collector Template

Overview

Architecture

What Gets Created

Prerequisites

Configuration

Mode and Replicas

Simple Mode

Metrics Ingestion

Ingestion Auth

Access

Advanced Mode

Setting Up Tracing

Connecting

Important Notes

External References