Datadog

Overview

Datadog is a robust and comprehensive monitoring service for cloud-scale applications, providing full visibility into IT infrastructure. The service combines metrics and events from servers, databases, applications, tools, and services to present a unified view of an entire stack. Datadog is capable of ingesting, analyzing, and visualizing logs from a variety of sources, including servers. This feature is a part of Datadog’s Log Management solution, which provides essential capabilities such as real-time log tailing and filtering, log analytics, and detailed visualizations. It can handle a wide range of log formats, including but not limited to, JSON, syslog, and common application logs. Datadog offers capabilities for log retention and archiving, enabling compliance with various regulatory standards. It also supports role-based access controls to secure your logs and control who can access what information. Follow the steps below to configure log shipping to Datadog.

Step 1 - Credential Procurement

An API key is required to ship logs to Datadog. Follow these steps to obtain the API key, store it as an Opaque Secret, and configure external logging.

From the Datadog dashboard, hover over your username at the bottom of the left menu and click Organization Settings.
In the middle menu, click API Keys.
Click the New Key button in the upper right corner, enter a key name, and click Create Key.
Click the Copy Key button. This will copy the key to your clipboard. Click the X to close the modal.
From the Control Plane Console UI, click Secrets from the left menu.
Click the New button.
Enter a Name for the secret, and select Opaque from the Secret Type list.
Paste the string from step #4 into the content text box and click Create.
This secret will be used when configuring logging using the UI Console or CLI.

Step 2 - Configure External Logging

External logging can be configured by using either the UI Console or CLI.

Enable Logging using the UI Console

From the Control Plane Console UI, click on Org in the left menu.
Click External Logs in the middle context menu.
Select Datadog and fill out the required fields.
Select the Opaque secret created to authenticate to Datadog. Refer to the credential procurement section to obtain and configure the necessary credentials.
Click Save.
After the configuration is complete, log entries will be available at Datadog within a few minutes.

The hostname will be similar to the domain name that resolves when using the Datadog dashboard. (i.e., The host http-intake.logs.us3.datadoghq.com maps to the dashboard domain us3.datadoghq.com)

Enable Logging using the CLI

The external logging configuration can be created / updated using the CLI’s cpln org patch ORG_NAME -f FILE.yaml command. Below is an example of an Org manifest (in YAML). Edit and save the YAML as a file and use it as an input to the CLI’s cpln org patch ORG_NAME -f FILE.yaml command. Refer to the credential procurement section to obtain and configure the necessary credentials.

Substitute: ORG_NAME, OPAQUE_SECRET, and possibly the host.

Use the host URL that matches your Datadog account. The host will be a similar domain name that resolves when using the Datadog dashboard.

YAML

kind: org
name: ORG_NAME
spec:
  logging:
    datadog:
      host: http-intake.logs.us3.datadoghq.com
      credentials: //secret/OPAQUE_SECRET

Metrics and Traces

Control Plane ships logs to Datadog natively. Metrics and traces take a different route: you run an OpenTelemetry Collector workload that holds your Datadog API key and forwards both signals.

Metrics — the collector scrapes your org’s Prometheus-compatible federation endpoint at metrics.cpln.io.
Traces — the GVC’s OpenTelemetry tracing provider sends spans to the collector over cpln.local.

spec.tracing.provider.otel accepts an endpoint only. It carries no headers or credentials, so traces cannot be sent directly to a Datadog intake, which requires an API key. The collector holds the key and adds it when forwarding.

Create the Secrets

Store your Datadog API key as an Opaque Secret, as described in Credential Procurement. The examples below call it datadog-api-key.
Follow the Export Metrics to Prometheus guide to create a service account, generate a key, and grant the service account readMetrics with a policy.
Store that service account key as a second Opaque Secret. The examples below call it cpln-metrics-token.

Deploy the Collector

The manifest below creates the collector configuration (as an Opaque Secret), an identity and policy that let the workload reveal its secrets, and the collector workload.

Substitute: ORG_NAME, GVC_NAME, and the Datadog site for your account (datadoghq.com, us3.datadoghq.com, us5.datadoghq.com, or datadoghq.eu).
If your account is on datadoghq.eu, replace *.datadoghq.com with *.datadoghq.eu in the firewall.
Adjust the match[] selector to the series you need.

YAML

kind: secret
name: datadog-otel-config
type: opaque
data:
  encoding: plain
  payload: |
    extensions:
      health_check:
        endpoint: 0.0.0.0:13133
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
      prometheus:
        config:
          scrape_configs:
            - job_name: cpln-federate
              scrape_interval: 1m
              honor_labels: true
              scheme: https
              metrics_path: /metrics/org/ORG_NAME/api/v1/federate
              params:
                'match[]':
                  - '{__name__=~".+"}'
              authorization:
                type: Bearer
                credentials: ${env:CPLN_METRICS_TOKEN}
              static_configs:
                - targets:
                    - metrics.cpln.io
    processors:
      batch: {}
    connectors:
      datadog/connector: {}
    exporters:
      datadog/exporter:
        api:
          site: datadoghq.com
          key: ${env:DD_API_KEY}
    service:
      extensions: [health_check]
      pipelines:
        traces:
          receivers: [otlp]
          processors: [batch]
          exporters: [datadog/connector, datadog/exporter]
        metrics:
          receivers: [prometheus, datadog/connector]
          processors: [batch]
          exporters: [datadog/exporter]
---
kind: identity
name: datadog-otel-collector
gvc: GVC_NAME
---
kind: policy
name: datadog-otel-collector
targetKind: secret
targetLinks:
  - //secret/datadog-otel-config
  - //secret/datadog-api-key
  - //secret/cpln-metrics-token
bindings:
  - permissions:
      - reveal
    principalLinks:
      - //gvc/GVC_NAME/identity/datadog-otel-collector
---
kind: workload
name: datadog-otel-collector
gvc: GVC_NAME
spec:
  type: standard
  identityLink: //identity/datadog-otel-collector
  containers:
    - name: collector
      image: otel/opentelemetry-collector-contrib:0.157.0
      cpu: 200m
      memory: 256Mi
      args:
        - '--config=/etc/otel-collector/config.yaml'
      env:
        - name: DD_API_KEY
          value: cpln://secret/datadog-api-key.payload
        - name: CPLN_METRICS_TOKEN
          value: cpln://secret/cpln-metrics-token.payload
      ports:
        - number: 4317
          protocol: grpc
      volumes:
        - path: /etc/otel-collector/config.yaml
          uri: cpln://secret/datadog-otel-config.payload
      readinessProbe:
        httpGet:
          path: /
          port: 13133
  defaultOptions:
    autoscaling:
      metric: disabled
      minScale: 1
      maxScale: 1
  firewallConfig:
    external:
      outboundAllowHostname:
        - metrics.cpln.io
        - '*.datadoghq.com'
    internal:
      inboundAllowType: same-gvc

Save the manifest as a file and apply it with cpln apply --file FILE.yaml --org ORG_NAME. See cpln apply.

Every replica scrapes the federation endpoint. If the GVC has more than one location, each location runs a replica and metrics are sent to Datadog once per location. To avoid duplicates, run the prometheus receiver in a separate collector in a single-location GVC, and keep only the otlp receiver in the collector that receives traces.

Egress charges apply to metrics scraped from Control Plane, and Datadog bills each federated series as a custom metric. Narrow the match[] selector to the series you need.

Point Tracing at the Collector

Set the GVC’s tracing provider to OpenTelemetry with the collector’s internal address, as described in the OpenTelemetry tracing provider reference. Apply it with cpln gvc patch GVC_NAME -f FILE.yaml. Changing the tracing configuration restarts all workloads in the GVC.

YAML

kind: gvc
name: GVC_NAME
spec:
  tracing:
    provider:
      otel:
        endpoint: datadog-otel-collector.GVC_NAME.cpln.local:4317
    sampling: 10

sampling is the percentage of requests that are traced. Metrics appear in Datadog’s Metrics Explorer and traces in APM within a few minutes. If nothing arrives, check the collector workload’s logs for exporter or scrape errors.

The OTel Collector template also deploys a collector you can point tracing at. Its advanced mode accepts a full collector configuration, including the datadog exporter, but it does not inject environment variables, so the API key would have to be written into the configuration itself.

Getting Started

Quickstart

Concepts

Core

Reference

Overview

Step 1 - Credential Procurement

Step 2 - Configure External Logging

Enable Logging using the UI Console

Enable Logging using the CLI

Metrics and Traces

Create the Secrets

Deploy the Collector

Point Tracing at the Collector

​Overview

​Step 1 - Credential Procurement

​Step 2 - Configure External Logging

​Enable Logging using the UI Console

​Enable Logging using the CLI

​Metrics and Traces

​Create the Secrets

​Deploy the Collector

​Point Tracing at the Collector

Overview

Step 1 - Credential Procurement

Step 2 - Configure External Logging

Enable Logging using the UI Console

Enable Logging using the CLI

Metrics and Traces

Create the Secrets

Deploy the Collector

Point Tracing at the Collector