Airflow

​

Overview

​

What Gets Created

• GVC — A dedicated GVC across the specified locations.
• Airflow Webserver — The Airflow web UI for managing DAGs, monitoring task execution, and viewing logs.
• Celery Workers — Distributed task execution workers that process DAG tasks.
• Redis — A Redis broker for the Celery task queue, with persistent storage.
• PostgreSQL — A PostgreSQL database for Airflow metadata storage.
• Volume Sets — Persistent storage for Airflow DAG data, PostgreSQL, and Redis.
• KEDA ScaledObject (optional) — Automatically scales Celery workers up or down based on Redis queue length.
• Secret — A dictionary secret containing the PostgreSQL credentials, JWT signing key, Fernet encryption key, and admin password, shared across all Airflow workloads.
• Identity & Policy — An identity bound to the workloads with reveal access to the Airflow configuration secret.

​

Pre-Deployment Checklist

​

Installation

​

Configuration

# Global Virtual Cloud (GVC) settings
gvc:
  name: airflow
  locations:
    - name: aws-eu-central-1

# Postgres database configuration
postgres:
  image: postgres:18
  resources:
    minCpu: 250m
    maxCpu: 500m
    minMemory: 512Mi
    maxMemory: 1024Mi
  config:
    username: username
    password: password
    database: airflow
  volumeset:
    capacity: 10 # initial capacity in GiB (minimum is 10)

# Redis cache configuration
redis:
  image: redis:7.4
  resources:
    cpu: 250m
    memory: 512Mi
  volumeset:
    capacity: 10 # initial capacity in GiB (minimum is 10)

# Apache Airflow configuration
airflow:
  webserver:
    image: apache/airflow:3.0.3
    resources:
      cpu: 2000m
      memory: 3Gi
  celeryWorker:
    image: controlplanecorporation/celery:v1
    resources:
      cpu: 256m
      memory: 512Mi
  webPort: 8080 # Port for accessing the Airflow web interface

  auth:
    jwtSecret: CHANGE_ME # REQUIRED: generate with "openssl rand -base64 48"
    jwtExpirationDelta: 3600 # JWT token expiration time in seconds
    jwtRefreshThreshold: 300 # Threshold before token expires to allow refresh (seconds)
    fernetKey: CHANGE_ME # REQUIRED: generate with "python3 -c 'from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())'"

  admin:
    username: admin
    password: CHANGE_ME # REQUIRED: change before deploying to production

  scheduler:
    dagDirListInterval: 10 # How often to check DAG folder (seconds)
    minFileProcessInterval: 10 # Minimum interval to process DAG files (seconds)

  celery:
    workerConcurrency: 1 # Number of tasks each worker can run concurrently

volumeset:
  airflow:
    capacity: 10 # initial capacity in GiB (minimum is 10)

# Firewall configuration
firewallConfig:
  inboundAllowCIDR:
    - 0.0.0.0/0 # Restrict to specific IPs in production (e.g. - 203.0.113.0/24)

# Git-sync configuration for DAG delivery
gitSync:
  enabled: false
  repo: "" # Git repository URL (e.g. https://github.com/org/dags)
  branch: main # Branch to sync
  period: 60s # How often to sync
  subPath: "" # Optional subfolder within the repo containing DAGs
  auth:
    token: "" # Personal access token for private repos (leave empty for public repos)

# KEDA (Kubernetes Event-driven Autoscaling) configuration
# NOT SUPPORTED in gcp/us-central1
keda:
  enabled: true # Enable or disable KEDA autoscaling
  minScale: 1 # Minimum number of Celery workers
  maxScale: 3 # Maximum number of Celery workers
  scaleToZeroDelay: 300 # Time before scaling to zero (seconds)
  listLength: 3 # Queue length threshold to trigger scaling
  cooldownPeriod: 1 # Cooldown between scaling events (seconds)
  initialCooldownPeriod: 1 # Cooldown after startup before scaling (seconds)
  pollingInterval: 4 # Interval at which KEDA queries metrics (seconds)

​

GVC

• gvc.name — The name of the GVC. Must be unique per deployment.
• gvc.locations — List of cloud locations to deploy to (e.g., aws-eu-central-1).

​

PostgreSQL

• postgres.image — PostgreSQL Docker image.
• postgres.resources — CPU and memory bounds for the PostgreSQL workload (minCpu, maxCpu, minMemory, maxMemory).
• postgres.config.username / postgres.config.password — Database credentials. Change the default password before deploying to production.
• postgres.config.database — Name of the Airflow metadata database (default: airflow).
• postgres.volumeset.capacity — Persistent storage for PostgreSQL data (GiB, minimum 10).

​

Redis

• redis.image — Redis Docker image.
• redis.resources — CPU and memory allocated to Redis.
• redis.volumeset.capacity — Persistent storage for Redis data (GiB, minimum 10).

​

Airflow Webserver and Workers

• airflow.webserver.image / airflow.celeryWorker.image — Docker images for the webserver and Celery workers.
• airflow.webserver.resources / airflow.celeryWorker.resources — CPU and memory per component.
• airflow.webPort — Port the Airflow web UI listens on (default 8080).

​

Authentication

• airflow.auth.jwtSecret — Secret key used to sign JWT tokens for API authentication. Generate a secure value with:
  Copy

  Ask AI

  openssl rand -base64 48
  
• airflow.auth.fernetKey — Key used to encrypt stored connections and variables. Generate with:
  Copy

  Ask AI

  python3 -c 'from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())'
  
• airflow.auth.jwtExpirationDelta — Token lifetime in seconds (default 3600).
• airflow.auth.jwtRefreshThreshold — Seconds before expiry at which a token refresh is allowed (default 300).

​

Admin Account

• airflow.admin.username — Username for the initial Airflow admin account (default: admin).
• airflow.admin.password — Password for the initial admin account. Change before deploying to production.

​

Scheduler

• airflow.scheduler.dagDirListInterval — How often the scheduler scans the DAG folder for new or modified files (seconds).
• airflow.scheduler.minFileProcessInterval — Minimum interval between processing the same DAG file (seconds).

​

Celery

• airflow.celery.workerConcurrency — Number of tasks a single Celery worker can execute concurrently.

​

Storage

• volumeset.airflow.capacity — Persistent storage for the Airflow home directory shared across workloads (GiB, minimum 10).

​

Firewall

• firewallConfig.inboundAllowCIDR — List of CIDR ranges allowed to reach the Airflow webserver. Defaults to 0.0.0.0/0 (public). Restrict to specific IP ranges in production.

​

Git-Sync

​

KEDA Autoscaling

​

Connecting to Airflow

https://<gvc-name>-airflow-webserver.<gvc-name>.cpln.app

​

API Access

curl -X POST https://<your-airflow-url>/auth/token \
  -H "Content-Type: application/json" \
  -d '{"username": "admin", "password": "your-password"}'

curl https://<your-airflow-url>/api/v2/dags \
  -H "Authorization: Bearer <token>"

​

Production Considerations

• Change all CHANGE_ME values before deploying — jwtSecret, fernetKey, admin.password, and postgres.config.password are all required.
• Restrict firewallConfig.inboundAllowCIDR to trusted IP ranges to limit access to the Airflow UI.
• Enable git-sync for reliable, version-controlled DAG delivery.
• Auth: SimpleAuthManager is not recommended for deployments requiring enterprise SSO. Evaluate an OAuth/OIDC integration for those use cases.

​

External References