Export Metrics to Prometheus

​

Overview

This guide demonstrates how to export metrics from multiple organizations using the Prometheus /federate endpoint. This setup allows you to centralize metrics collection by federating Control Plane metrics into a single external Prometheus instance or service that can collect data using a Prometheus scrape job.

​

Prerequisites

• Superuser access to the Control Plane Console for each organization.
• A Prometheus instance configured to scrape metrics or a third party service that can collect metrics using a Prometheus scrape job.

​

Steps

​

In the Source Organization

1. Create a Service Account:

   • Log in to the Console.
   • Create a new Service Account named prometheus-federate.

2. Generate a Key:

   • Create a new key for the prometheus-federate Service Account and save it securely.

3. Create a Policy:

You can create the policy using either the Control Plane Console UI or by using YAML configuration.

Using the Control Plane Console UI:

• Create a policy named prometheus-federate that grants readMetrics permission to the Service Account.
• Navigate to the Policies section in the Control Plane Console.
• Click on New + and fill in the details:
  • Name: prometheus-federate
  • Description: prometheus-federate
  • Target Kind: org
• Click Next (Bindings):
• Click Add Binding:
  • On the Permissions tab, check readMetrics
  • On the Service Accounts tab, check prometheus-federate
• Click Add
• Click Create

Using YAML Configuration:

kind: policy
name: prometheus-federate
description: prometheus-federate
tags: {}
bindings:
  - permissions:
      - readMetrics
    principalLinks:
      - /org/org-2/serviceaccount/prometheus-federate
target: all
targetKind: org

​

In the Prometheus Configuration

1. Configure Scrape Job:

   • Add the following scrape configuration to your Prometheus instance:

   scrape_configs:
     - job_name: 'federate'
       scrape_interval: 1m
       honor_labels: true
       scheme: https
       metrics_path: '/metrics/org/${org}/api/v1/federate'
       params:
         'match[]':
           - '{__name__=~".+"}'  # Adjust this matcher as needed
       authorization:
         type: Bearer
         credentials: "${CPLN_SERVICE_ACCOUNT_TOKEN}"  # Token value from env variable
       static_configs:
         - targets:
             - 'metrics.cpln.io'
   

   • Replace ${org} with the actual organization name.
   • Ensure ${CPLN_SERVICE_ACCOUNT_TOKEN} is set as an environment variable containing the token from the prometheus-federate Service Account.

​

Verification

• After configuring the scrape job, verify that Prometheus is successfully collecting metrics from the source organization by checking the Prometheus UI.
• You should be able to see the same metrics in the Prometheus UI as you would in the Control Plane managed Grafana instance.

​

Additional Notes

• Adjust the match[] parameter in the scrape configuration to filter the metrics as needed.
• Repeat the steps for additional organizations by creating separate Service Accounts and policies.
• Egress charges apply to metrics being scraped from the Control Plane.

​

Cleanup

• Delete the scrape job in the Prometheus instance.
• Delete the Policy and Service Account in the source organization.