Autoscaling

Autoscaling Overview

Workload auto-scaling is configured by setting a strategy, a target value, and in some cases as metric percentile. Together these values determine when the workload will scale up & down. As the system scales up, traffic will not be sent to the new replicas until they pass the readiness probe, if configured. If there is no probe configured or if it is a basic TCP port check, the requests will hit the new replicas before they are ready to respond. This could cause a delay or errors for end-user traffic.

You can configure autoscaling in the default options for a workload (defaultOptions) and in any of the location-specific options.

Scaling Strategies

The scaling strategy is set using autoscaling.metric.

Disabled (disabled)
- Scaling will be disabled.
Concurrent Requests Quantity (concurrency)
- The average number of requests executing at a given point in time across all the replicas. (requests * requestDuration)/(timePeriod * replicas).
- Example: A workload with 5 replicas received 1000 requests with an average response time of 50ms (05 seconds) over a 1 second period. The concurrent requests metric for that period is (1000 * .05)/(1 * 5) = 10.
Requests Per Second (rps)
- The raw number of requests received by a workload each second divided by the number of replicas. Requests are counted even if they haven’t been completed.
Percentage of CPU Utilization (cpu)
- The percentage of CPU consumed by system and user processes in the container(s) as specified in the container cpu field.
Request Latency (latency)
- The request response time (at a configurable percentile) in milliseconds, averaged across all replicas.
Memory Utilization (memory)
- The percentage of memory consumed by system and user processes in the container(s) as specified in the container memory field.

Caveats when choosing a workload type and a scaling strategy:

Serverless workloads cannot use the latency scaling strategy or any of the Multi Metric scaling strategies.
Standard workloads cannot use the concurrency scaling strategy

The scale to zero functionality is available for Serverless workloads when using the rps or concurrency scaling strategies, and for Standard and Stateful workloads when using the keda metric. Cron workloads cannot scale to zero.

Autoscaling Target

The target is the value that the system will try to keep the chosen scaling metric (such as concurrency, rps, cpu, latency, or memory) near but below.

Autoscaling Standard Workloads

For standard workloads, Control Plane runs two asynchronous control loops:

The Scaling Decision Loop
The Metric Calculation Loop

Because of this asynchronous structure, autoscaling decisions may be made based on a metric value that is as old as the metric’s collection rate (usually 20 seconds).

The Scaling Decision Loop

A workload’s scale is evaluated every 15 seconds, using the value most recently calculated by the Metric Calculation Loop. Each time an evaluation is made the chosen metric is averaged across all available replicas and compared against the scale target. When scaling up, Control Plane does not enforce a stabilization window; the number of pods will increase as soon as the scaling algorithm dictates. When scaling down, a stabilization window of 5 minutes is used; the highest number of pods recommended by the scaling algorithm within the past 5 minutes will be applied to the running workload.

The Metric Calculation Loop

Requests per Second

Every 20 seconds, Control Plane calculates the average number of requests per second over the past 60 seconds.

Latency

Every 20 seconds, Control Plane calculates latency, using the response time of the workload once requests are received, using an average over the past 60 seconds at the specified percentile (p50, p75, p99).

CPU

Every 15 seconds, Control Plane calculates the average CPU usage over the past 15 seconds.

Memory

Every 15 seconds, Control Plane calculates the average memory usage over the past 15 seconds.

Autoscaling Serverless Workloads

The current capacity is evaluated every 2 seconds and compared against the scale target. It averages requests completed over the previous 60 seconds to avoid rapid changes. If ever a scaling decision is made which results in a scale increase above 200% then it suspends scale down decisions and averages over 6 seconds for 60 seconds. This is to allow for rapid scaling when a burst of traffic is detected.

Special considerations for the latency scaling strategyBecause request latency is represented as a distribution, when using the latency scaling strategy, you must choose a metric percentile by setting the autoscaling.metricPercentile property to one of the following values:

p50
p75
p99

Autoscaling Options

Minimum Scale (autoscaling.minScale)
- The minimum allowed number of replicas.
- Control Plane can scale the workload down to 0 when there is no traffic and scale up immediately to fulfill new requests.
- Must be between 0 and Maximum Scale inclusive.
Maximum Scale (autoscaling.maxScale)
- The maximum allowed number of replicas.
Scale to Zero Delay (autoscaling.scaleToZeroDelay)
- The amount of time (in seconds) with no requests received before a workload is scaled down to 0.
- Must be between 30 and 3600 inclusive.
Maximum Concurrency (autoscaling.maxConcurrency)
- A hard maximum for the number of concurrent requests allowed to a replica.
- If no replicas are available to fulfill the request, it will be queued until a replica with capacity is available and delivered as soon as one is available again.
- Capacity can be available from requests completing or when a new replica is available from scale out.
- A value of 0 allows all requests.
- Must be between 0 and 30000 inclusive.
Metric (autoscaling.metric)
- Controls the metric which will be used for scaling decisions. The goal is to maintain the target across all replicas of a deployment. Options include:
  - concurrency: Uses the number of concurrent requests for the target.
  - cpu: Uses % processor time for the target.
  - memory: Uses memory in Mi for the target.
  - rps: Uses requests per second for the target.
  - latency: Uses the average request response time for the target. Not available for Serverless workloads.
Multi Metric (autoscaling.multi)
- Allows specifying multiple metrics for autoscaling decisions.
- Each metric must be unique and is defined with a target value.
- Not available for Serverless workloads.
Metric Percentile (autoscaling.metricPercentile)
- The latency metric is represented as a distribution, so a percentile within the distribution must be chosen to be used with the target.
- The default value is p50.
- Control Plane supports p50, p75, and p99 metric percentiles.
- For example, if the percentile is p50 and the target is 100, when the 50th percentile of latency is greater than 100ms for the workload, a scale-up decision will be made.

Capacity AI is not available if CPU Utilization is selected because dynamic allocation of CPU resources cannot be accomplished while scaling replicas based on the usage of its CPU. Additionally, Capacity AI may not be enabled when multi metric is set. See Capacity AI Restrictions for the full list.

Autoscaling Examples

Concurrency

spec:
  defaultOptions:
    autoscaling:
      metric: concurrency
      maxConcurrency: 0
      maxScale: 1
      minScale: 5
      scaleToZeroDelay: 300
      target: 100

CPU

spec:
  defaultOptions:
    autoscaling:
      metric: cpu
      target: 80
      minScale: 1
      maxScale: 5

Memory

spec:
  defaultOptions:
    autoscaling:
      metric: memory
      target: 80
      minScale: 1
      maxScale: 5

RPS

spec:
  defaultOptions:
    autoscaling:
      metric: rps
      target: 100
      minScale: 1
      maxScale: 5

Latency

spec:
  defaultOptions:
    autoscaling:
      metric: latency
      target: 100
      minScale: 1
      maxScale: 5
      metricPercentile: p50

Multi Metric

spec:
  defaultOptions:
    autoscaling:
      minScale: 1
      maxScale: 5
      multi:
        - metric: cpu
          target: 80
        - metric: memory
          target: 80

Keda

In cases where the autoscaling strategies provided by Control Plane do not meet your needs, you can use Keda to scale workloads based on custom metrics. Keda is a Kubernetes-based event-driven autoscaler that allows you to define custom scaling rules and metrics. A workload can use keda to scale based on custom metrics, like redis queue lengths or kafka topic lag. It is available for both standard and stateful workloads. In order to use keda, you must first enable it on the gvc

If Keda requires network or cloud resources, you must attach a valid identity on the keda configuration in the gvc

Keda Configuration

spec:
  defaultOptions:
    autoscaling:
      metric: keda
      keda:
        triggers:
          - type: redis
            metadata:
              address: my-redis.my-gvc.cpln.local:6379
              queueLength: '5'
              passwordFromEnv: REDIS_PASSWORD
              usernameFromEnv: REDIS_USERNAME

Each entry under triggers accepts:

type (required): The KEDA trigger type, e.g. redis, prometheus, aws-sqs-queue.
metadata (optional): The configuration parameters the trigger type requires. Consult the KEDA scaler documentation for the parameter names of your trigger type.
name (optional): A name for the trigger. If omitted, Control Plane generates one from the trigger type.
metricType (optional): AverageValue, Value, or Utilization. Overrides the metric type KEDA reports for this trigger.
useCachedMetrics (optional): When true, KEDA reuses the metric value collected in the last polling interval instead of re-querying the source on every check.
authenticationRef (optional): References a GVC keda secret by name. See TriggerAuthentication.

Internal access

If keda requires access to a Control Plane workload, that workload must be configured to allow keda access.

spec:
  firewallConfig:
    internal:
      inboundAllowWorkload:
        - cpln://internal/keda

Advanced Options

For a more advanced setup, perhaps utilizing multiple triggers or additional configurations, you can define the keda advanced section in the workload spec:

spec:
  defaultOptions:
    autoscaling:
      metric: keda
      keda:
        triggers: ...
        advanced:
          scalingModifiers:
            activationTarget: '0'
            formula: >-
              (latency > 3000) ? (existing_pods) : min(max((queue_length /
              15) + 1, 1), existing_pods * 1.15 + 1, existing_pods + 7)
            metricType: AverageValue
            target: '1'

HPA scaling behavior

Keda creates a Kubernetes HorizontalPodAutoscaler for the workload. Use advanced.horizontalPodAutoscalerConfig.behavior to tune how that HPA scales in each direction, for example to slow down scale-down and avoid flapping, cap scale-up bursts, or disable scaling in one direction. The block maps directly to the Kubernetes HPA behavior field. scaleUp and scaleDown are both optional and accept the same fields:

stabilizationWindowSeconds: Number of seconds past recommendations are considered before scaling. 0 to 3600.
selectPolicy: Which policy wins when several apply. Max (default), Min, or Disabled to turn off scaling in this direction.
policies: One or more scaling policies. Each has a type of Pods or Percent, a value greater than 0, and a periodSeconds between 1 and 1800.

spec:
  defaultOptions:
    autoscaling:
      metric: keda
      keda:
        triggers: ...
        advanced:
          horizontalPodAutoscalerConfig:
            behavior:
              scaleUp:
                stabilizationWindowSeconds: 0
                selectPolicy: Max
                policies:
                  - type: Percent
                    value: 100
                    periodSeconds: 60
                  - type: Pods
                    value: 4
                    periodSeconds: 60
              scaleDown:
                stabilizationWindowSeconds: 300
                selectPolicy: Min
                policies:
                  - type: Percent
                    value: 25
                    periodSeconds: 60
                  - type: Pods
                    value: 2
                    periodSeconds: 60

The name of the HPA is managed by Control Plane. The keda horizontalPodAutoscalerConfig.name field is not configurable.

Polling and Cooldown

Tune how often KEDA checks triggers and how long it waits before scaling back down:

spec:
  defaultOptions:
    autoscaling:
      metric: keda
      keda:
        triggers: ...
        pollingInterval: 30
        cooldownPeriod: 300
        initialCooldownPeriod: 10

pollingInterval: How often, in seconds, KEDA checks each trigger.
cooldownPeriod: How long, in seconds, KEDA waits after a trigger last reported active before scaling back down to minScale.
initialCooldownPeriod: How long, in seconds, KEDA waits after the workload starts before the cooldown period begins.

All three are optional and, if set, must be 1 or greater. When omitted, KEDA applies its own defaults. See the KEDA ScaledObject spec for details.

Fallback

When a trigger’s metric source becomes unreachable, KEDA can fall back to a fixed replica count instead of leaving the workload at its last known scale:

spec:
  defaultOptions:
    autoscaling:
      metric: keda
      keda:
        triggers:
          - type: redis
            metadata:
              address: my-redis.my-gvc.cpln.local:6379
              queueLength: '5'
        fallback:
          failureThreshold: 3
          replicas: 2
          behavior: static

failureThreshold (required): Number of consecutive failures to retrieve a trigger’s metric before fallback engages.
replicas (required): Number of replicas to scale to while fallback is active.
behavior (optional): static, currentReplicas, currentReplicasIfHigher, or currentReplicasIfLower. Controls how replicas is applied relative to the workload’s current replica count. Defaults to static in KEDA when omitted.

fallback requires at least one trigger, and at least one of those triggers must resolve to metric type AverageValue (the default when a trigger doesn’t set metricType) on a type other than cpu or memory.

TriggerAuthentication

If your keda trigger requires authentication, store the credentials in a dictionary secret and add it to the gvc keda configuration. For every secret listed under spec.keda.secrets, Control Plane creates a KEDA TriggerAuthentication object named after the secret. Each key of the dictionary becomes an authentication parameter with the same name, so the keys must exactly match the parameter names the scaler expects. Consult the KEDA scaler documentation for the parameter names of your trigger type.

Only secrets of type dictionary can be used as TriggerAuthentication objects. Store values as plain text; Control Plane handles the encoding.

For example, a dictionary secret named kafka-creds with the keys username and password added to the gvc:

spec:
  keda:
    enabled: true
    secrets:
      - //secret/kafka-creds

becomes a TriggerAuthentication named kafka-creds that provides the username and password parameters. A workload trigger references it by the secret name:

spec:
  defaultOptions:
    autoscaling:
      metric: keda
      keda:
        triggers:
          - type: kafka
            metadata:
              bootstrapServers: kafka.my-gvc.cpln.local:9092
              consumerGroup: my-group
              topic: test-topic
            authenticationRef:
              name: kafka-creds

Example: Prometheus with a bearer token

The KEDA prometheus scaler expects a parameter named bearerToken when using bearer authentication.

Create a dictionary secret with a bearerToken key holding the token. If your Prometheus server uses a private CA, add its PEM under a ca key:

kind: secret
name: prom-bearer
type: dictionary
data:
  bearerToken: '<token>'

Add the secret to the gvc keda configuration:

spec:
  keda:
    enabled: true
    secrets:
      - //secret/prom-bearer

Reference it from the workload trigger and set authModes: bearer:

spec:
  defaultOptions:
    autoscaling:
      metric: keda
      keda:
        triggers:
          - type: prometheus
            metadata:
              serverAddress: https://prometheus.example.com
              query: sum(rate(http_requests_total[2m]))
              threshold: '100'
              authModes: bearer
            authenticationRef:
              name: prom-bearer

Example: Amazon SQS queue length

Scale a worker workload on the number of messages waiting in an SQS queue. This uses the KEDA AWS SQS Queue scaler and the identity attached to the GVC keda configuration, so no AWS keys are stored.

Create an identity in the GVC with cloud access to the queue. The scaler only calls sqs:GetQueueAttributes, so the AWS managed AmazonSQSReadOnlyAccess policy is enough:

kind: identity
name: keda-identity
gvc: my-gvc
aws:
  cloudAccountLink: //cloudaccount/my-aws-account
  policyRefs:
    - aws::AmazonSQSReadOnlyAccess

Enable keda on the GVC and attach the identity:

spec:
  keda:
    enabled: true
    identityLink: //identity/keda-identity

Configure the workload. metric must be keda, and target must not be set:

spec:
  defaultOptions:
    autoscaling:
      metric: keda
      minScale: 0
      maxScale: 20
      keda:
        triggers:
          - type: aws-sqs-queue
            metadata:
              queueURL: https://sqs.us-east-1.amazonaws.com/123456789012/my-queue
              queueLength: '5'
              awsRegion: us-east-1
              identityOwner: operator

KEDA targets queueLength messages per replica, so 100 messages in the queue (visible plus in flight) scale the workload to 20 replicas, capped by maxScale. With minScale: 0 the workload scales to zero when the queue is empty and back up when messages arrive.

identityOwner: operator makes the scaler use the keda operator’s credentials, which come from the identity attached to the GVC keda configuration. Without it, KEDA expects access keys and the trigger fails with awsAccessKeyID not found. To use access keys instead, omit identityOwner, store awsAccessKeyID and awsSecretAccessKey in a dictionary secret, and reference it with authenticationRef.

Example: Schedule-based scaling

Use the KEDA cron scaler to hold a workload at a set number of replicas during a time window, for example during business hours:

spec:
  defaultOptions:
    autoscaling:
      metric: keda
      minScale: 1
      maxScale: 10
      keda:
        triggers:
          - type: cron
            metadata:
              timezone: America/New_York
              start: 0 8 * * 1-5
              end: 0 18 * * 1-5
              desiredReplicas: '5'

Between start and end the workload runs desiredReplicas replicas. Outside the window it returns to minScale. minScale must be less than or equal to maxScale, and desiredReplicas should fall between them. Set minScale: 0 to scale to zero outside the window.

You can combine a cron trigger with other triggers, for example a queue or Prometheus trigger. KEDA uses the highest replica count requested by any trigger.

Prometheus metrics

You can configure keda to use prometheus metrics for autoscaling. First, follow these steps to create a service account with access to readMetrics. Then, configure the keda trigger with the service account token:

spec:
  defaultOptions:
    autoscaling:
      metric: keda
      keda:
        triggers:
          - type: prometheus
            metadata:
              query: >
                (histogram_quantile(90/100 , sum(avg_over_time(request_duration_ms_bucket{workload="workload-name", gvc="gvc-name"}[1m])) by(gvc,workload, le)))
              threshold: '1'
              customHeaders: >-
                Authorization=Bearer <service-account-token>
              serverAddress: https://metrics.cpln.io:443/metrics/org/<org-name>
              activationThreshold: '0'
            name: workload-latency

Instead of inlining the token with customHeaders, store it in a dictionary secret under a bearerToken key and reference it via a TriggerAuthentication.

Query Examples

You can customize the PromQL query to use different metrics and calculations: The request_duration_ms_bucket histogram is already a rate per second, so average it over a window with avg_over_time rather than wrapping it in rate(). P99 Latency:

histogram_quantile(0.99,
  sum(avg_over_time(request_duration_ms_bucket{
    gvc="my-gvc",
    workload="target-workload"
  }[5m])) by (le)
)

Median Latency:

histogram_quantile(0.5,
  sum(avg_over_time(request_duration_ms_bucket{
    gvc="my-gvc",
    workload="target-workload"
  }[5m])) by (le)
)

Container Restarts (scale when restarts exceed threshold):

sum(rate(container_restarts{
  gvc="my-gvc",
  workload="problematic-service"
}[5m]))

Ready Replicas (scale when ready replicas fall below expected):

sum(workload_ready_replicas{
  gvc="my-gvc",
  workload="backend-service"
})

Requests Per Second (across all replicas):

sum(requests_per_second{
  gvc="my-gvc",
  workload="api-gateway"
})

Custom Metrics: If your workload exposes custom Prometheus metrics (see Custom Metrics documentation), you can use them for scaling:

# Example: Scale based on queue depth
sum(my_app_queue_depth{
  gvc="my-gvc",
  workload="queue-processor"
})

# Example: Scale based on cache hit ratio
sum(rate(my_app_cache_misses_total{
  gvc="my-gvc",
  workload="cache-service"
}[5m])) /
sum(rate(my_app_cache_requests_total{
  gvc="my-gvc",
  workload="cache-service"
}[5m]))

How It Works

Metrics Collection: Control Plane automatically collects default metrics for all workloads (latency, requests, replicas, etc.) and can be configured to collect Custom Metrics you expose from your application
KEDA Polling: KEDA polls the Prometheus endpoint at metrics.cpln.io using the service account token for authentication
Query Evaluation: The PromQL query is evaluated against the metrics to calculate the current value
Scaling Decision: When the metric value exceeds the threshold, KEDA triggers a scale-up event; when it falls below, KEDA scales down

Additional Resources

Default Metrics - List of available metrics
KEDA Prometheus Scaler Documentation

Getting Started

Quickstart

Concepts

Core

Reference

Autoscaling Overview

Scaling Strategies

Autoscaling Target

Autoscaling Standard Workloads

The Scaling Decision Loop

The Metric Calculation Loop

Requests per Second

Latency

CPU

Memory

Autoscaling Serverless Workloads

Autoscaling Options

Autoscaling Examples

Concurrency

CPU

Memory

RPS

Latency

Multi Metric

Keda

Keda Configuration

Internal access

Advanced Options

HPA scaling behavior

Polling and Cooldown

Fallback

TriggerAuthentication

Example: Prometheus with a bearer token

Example: Amazon SQS queue length

Example: Schedule-based scaling

Prometheus metrics

Query Examples

How It Works

Additional Resources

​Autoscaling Overview

​Scaling Strategies

​Autoscaling Target

​Autoscaling Standard Workloads

​The Scaling Decision Loop

​The Metric Calculation Loop

​Requests per Second

​Latency

​CPU

​Memory

​Autoscaling Serverless Workloads

​Autoscaling Options

​Autoscaling Examples

​Concurrency

​CPU

​Memory

​RPS

​Latency

​Multi Metric

​Keda

​Keda Configuration

​Internal access

​Advanced Options

​HPA scaling behavior

​Polling and Cooldown

​Fallback

​TriggerAuthentication

​Example: Prometheus with a bearer token

​Example: Amazon SQS queue length

​Example: Schedule-based scaling

​Prometheus metrics

​Query Examples

​How It Works

​Additional Resources

Autoscaling Overview

Scaling Strategies

Autoscaling Target

Autoscaling Standard Workloads

The Scaling Decision Loop

The Metric Calculation Loop

Requests per Second

Latency

CPU

Memory

Autoscaling Serverless Workloads

Autoscaling Options

Autoscaling Examples

Concurrency

CPU

Memory

RPS

Latency

Multi Metric

Keda

Keda Configuration

Internal access

Advanced Options

HPA scaling behavior

Polling and Cooldown

Fallback

TriggerAuthentication

Example: Prometheus with a bearer token

Example: Amazon SQS queue length

Example: Schedule-based scaling

Prometheus metrics

Query Examples

How It Works

Additional Resources