​
Overview

This guide demonstrates how to export metrics from multiple organizations using the Prometheus /federate endpoint. This setup allows you to centralize metrics collection by federating Control Plane metrics into a single external Prometheus instance or service that can collect data using a Prometheus scrape job.

​
Prerequisites

  • Superuser access to the Control Plane Console for each organization.
  • A Prometheus instance configured to scrape metrics or a third party service that can collect metrics using a Prometheus scrape job.

​
Steps

​
In the Source Organization

  1. Create a Service Account:
    • Log in to the Console.
    • Create a new Service Account named prometheus-federate.
  2. Generate a Key:
    • Create a new key for the prometheus-federate Service Account and save it securely.
  3. Create a Policy:
You can create the policy using either the Control Plane Console UI or by using YAML configuration. Using the Control Plane Console UI:
  • Create a policy named prometheus-federate that grants readMetrics permission to the Service Account.
  • Navigate to the Policies section in the Control Plane Console.
  • Click on New + and fill in the details:
    • Name: prometheus-federate
    • Description: prometheus-federate
    • Target Kind: org
  • Click Next (Bindings):
  • Click Add Binding:
    • On the Permissions tab, check readMetrics
    • On the Service Accounts tab, check prometheus-federate
  • Click Add
  • Click Create
Using YAML Configuration: 
kind: policy
name: prometheus-federate
description: prometheus-federate
tags: {}
bindings:
  - permissions:
      - readMetrics
    principalLinks:
      - /org/org-2/serviceaccount/prometheus-federate
target: all
targetKind: org

​
In the Prometheus Configuration

  1. Configure Scrape Job:
    • Add the following scrape configuration to your Prometheus instance:
    scrape_configs:
  - job_name: 'federate'
    scrape_interval: 1m
    honor_labels: true
    scheme: https
    metrics_path: '/metrics/org/${org}/api/v1/federate'
    params:
      'match[]':
        - '{__name__=~".+"}'  # Adjust this matcher as needed
    authorization:
      type: Bearer
      credentials: "${CPLN_SERVICE_ACCOUNT_TOKEN}"  # Token value from env variable
    static_configs:
      - targets:
          - 'metrics.cpln.io'
    • Replace ${org} with the actual organization name.
    • Ensure ${CPLN_SERVICE_ACCOUNT_TOKEN} is set as an environment variable containing the token from the prometheus-federate Service Account.

​
Verification

  • After configuring the scrape job, verify that Prometheus is successfully collecting metrics from the source organization by checking the Prometheus UI.
  • You should be able to see the same metrics in the Prometheus UI as you would in the Control Plane managed Grafana instance.

​
Additional Notes

  • Adjust the match[] parameter in the scrape configuration to filter the metrics as needed.
  • Repeat the steps for additional organizations by creating separate Service Accounts and policies.
  • Egress charges apply to metrics being scraped from the Control Plane.

​
Cleanup

  • Delete the scrape job in the Prometheus instance.
  • Delete the Policy and Service Account in the source organization.