> ## Documentation Index
> Fetch the complete documentation index at: https://docs.controlplane.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Airflow

> Deploy Apache Airflow on Control Plane using the Template Catalog. Covers configuration, volumes, scaling, and Celery workers with Redis broker and PostgreSQL metadata store.

## Overview

Apache Airflow is an open-source platform for programmatically authoring, scheduling, and monitoring workflows. This template deploys a full Airflow stack using CeleryExecutor, with Redis as the task queue broker and PostgreSQL as the metadata database. Celery workers can optionally be autoscaled using KEDA based on queue depth.

### What Gets Created

* **GVC** — A dedicated GVC across the specified locations.
* **Airflow Webserver** — The Airflow web UI for managing DAGs, monitoring task execution, and viewing logs.
* **Celery Workers** — Distributed task execution workers that process DAG tasks.
* **Redis** — A Redis broker for the Celery task queue, with persistent storage.
* **PostgreSQL** — A PostgreSQL database for Airflow metadata storage.
* **Volume Sets** — Persistent storage for Airflow DAG data, PostgreSQL, and Redis.
* **KEDA ScaledObject** (optional) — Automatically scales Celery workers up or down based on Redis queue length.
* **Secret** — A dictionary secret containing the PostgreSQL credentials, JWT signing key, Fernet encryption key, and admin password, shared across all Airflow workloads.
* **Identity & Policy** — An identity bound to the workloads with `reveal` access to the Airflow configuration secret.

## Pre-Deployment Checklist

Before deploying, generate and set the following required values in `values.yaml`:

| Value                      | How to generate                                                                              |
| -------------------------- | -------------------------------------------------------------------------------------------- |
| `airflow.auth.jwtSecret`   | `openssl rand -base64 48`                                                                    |
| `airflow.auth.fernetKey`   | `python3 -c 'from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())'` |
| `airflow.admin.password`   | Choose a strong password                                                                     |
| `postgres.config.password` | Choose a strong password                                                                     |

## Installation

This template has no external prerequisites. To install, follow the instructions for your preferred method:

<CardGroup cols={2}>
  <Card title="UI" href="/template-catalog/install-manage/ui" icon="laptop">
    Browse, install, and manage templates visually
  </Card>

  <Card title="CLI" href="/template-catalog/install-manage/cli" icon="terminal">
    Manage templates from your terminal
  </Card>

  <Card title="Terraform" href="/template-catalog/install-manage/terraform" icon={<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 128 128"><g fill-rule="evenodd"><path d="M77.941 44.5v36.836L46.324 62.918V26.082zm0 0" fill="#5c4ee5"/><path d="M81.41 81.336l31.633-18.418V26.082L81.41 44.5zm0 0" fill="#4040b2"/><path d="M11.242 42.36L42.86 60.776V23.941L11.242 5.523zm0 0M77.941 85.375L46.324 66.957v36.82l31.617 18.418zm0 0" fill="#5c4ee5"/></g></svg>}>
    Declare templates in your Terraform configurations
  </Card>

  <Card
    title="Pulumi"
    href="/template-catalog/install-manage/pulumi"
    icon={<svg xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 24 24" id="Pulumi-Icon--Streamline-Svg-Logos" height="24" width="24">
    <desc>
        Pulumi Icon Streamline Icon: https://streamlinehq.com
    </desc>
    <path fill="#f26e7e" d="M4.683025 13.3318c0.869125 -0.5018 0.870575 -2.1264 0.003225 -3.62865s-2.27504 -2.313275 -3.1441725 -1.811475C0.672945 8.3935 0.6715 10.0181 1.53885 11.52035c0.86735 1.502275 2.27505 2.313275 3.144175 1.81145Zm0.0052 3.2167c0.86735 1.502275 0.865925 3.126875 -0.003225 3.628675 -0.86915 0.5018 -2.2768275 -0.309225 -3.144175 -1.81145 -0.8673525 -1.50225 -0.8659075 -3.126875 0.003225 -3.628675 0.8691325 -0.5018 2.276825 0.309225 3.144175 1.81145Zm5.922875 3.4243c0.86735 1.50225 0.8659 3.126775 -0.003225 3.62875 -0.869125 0.501775 -2.27685 -0.309325 -3.1442 -1.81155 -0.867325 -1.50225 -0.865875 -3.12685 0.00325 -3.628675 0.869125 -0.5018 2.276825 0.309225 3.144175 1.811475Zm-0.001925 -6.845275c0.86735 1.50225 0.8659 3.12685 -0.003225 3.628675 -0.869125 0.5018 -2.276825 -0.309225 -3.144175 -1.811475 -0.86735 -1.50225 -0.8659 -3.12685 0.003225 -3.62865 0.869125 -0.501825 2.276825 0.3092 3.144175 1.81145Z" stroke-width="0.25"></path>
    <path fill="#8a3391" d="M22.45775 11.524125c0.86725 -1.502225 0.865925 -3.12685 -0.003225 -3.62865 -0.869125 -0.501825 -2.276825 0.3092 -3.144175 1.811475 -0.86735 1.50225 -0.8659 3.126825 0.003225 3.62865 0.869125 0.501825 2.276825 -0.3092 3.144175 -1.811475Zm0.000175 3.2151c0.869075 0.5018 0.870625 2.1264 0.003225 3.62865 -0.86735 1.50225 -2.27505 2.313275 -3.144175 1.81145 -0.869125 -0.5018 -0.870575 -2.126425 -0.003225 -3.62865 0.86735 -1.50225 2.27505 -2.313275 3.144175 -1.81145ZM16.536225 18.157875c0.86915 0.501825 0.8706 2.126425 0.00325 3.628675 -0.86735 1.502125 -2.275075 2.313225 -3.1442 1.81145 -0.869125 -0.50175 -0.870575 -2.126425 -0.003225 -3.62865 0.867375 -1.502275 2.27505 -2.3133 3.144175 -1.811475Zm-0.003325 -6.843775c0.869125 0.5018 0.870575 2.126425 0.003225 3.628675s-2.27505 2.313275 -3.1442 1.811475c-0.869125 -0.501825 -0.870575 -2.126425 -0.003225 -3.628675 0.86735 -1.502275 2.27505 -2.313275 3.1442 -1.811475Z" stroke-width="0.25"></path>
    <path fill="#f7bf2a" d="M15.138225 2.06721c0 1.003615 -1.40625 1.817215 -3.14095 1.817215 -1.7347 0 -3.14095 -0.8136 -3.14095 -1.817215C8.856325 1.06359 10.262575 0.25 11.997275 0.25c1.7347 0 3.14095 0.81359 3.14095 1.81721ZM9.2166 5.482375c0 1.003625 -1.40625 1.8172 -3.14095 1.8172 -1.7347 0 -3.14095 -0.813575 -3.14095 -1.8172s1.40625 -1.817225 3.14095 -1.817225c1.7347 0 3.14095 0.8136 3.14095 1.817225Zm8.71005 1.8172c1.7347 0 3.14095 -0.813575 3.14095 -1.8172s-1.40625 -1.817225 -3.14095 -1.817225c-1.7347 0 -3.14095 0.8136 -3.14095 1.817225s1.40625 1.8172 3.14095 1.8172Zm-2.788425 1.605625c0 1.003625 -1.40625 1.8172 -3.14095 1.8172 -1.7347 0 -3.14095 -0.813575 -3.14095 -1.8172 0 -1.0036 1.40625 -1.8172 3.14095 -1.8172 1.7347 0 3.14095 0.8136 3.14095 1.8172Z" stroke-width="0.25"></path>
    </svg>}
  >
    Declare templates in your Pulumi programs
  </Card>
</CardGroup>

## Configuration

The default `values.yaml` for this template:

```yaml theme={null}
# Global Virtual Cloud (GVC) settings
gvc:
  name: airflow
  locations:
    - name: aws-eu-central-1

# Postgres database configuration
postgres:
  image: postgres:18
  resources:
    minCpu: 250m
    maxCpu: 500m
    minMemory: 512Mi
    maxMemory: 1024Mi
  config:
    username: username
    password: password
    database: airflow
  volumeset:
    capacity: 10 # initial capacity in GiB (minimum is 10)

# Redis cache configuration
redis:
  image: redis:7.4
  resources:
    cpu: 250m
    memory: 512Mi
  volumeset:
    capacity: 10 # initial capacity in GiB (minimum is 10)

# Apache Airflow configuration
airflow:
  webserver:
    image: apache/airflow:3.0.3
    resources:
      cpu: 2000m
      memory: 3Gi
  celeryWorker:
    image: controlplanecorporation/celery:v1
    resources:
      cpu: 256m
      memory: 512Mi
  webPort: 8080 # Port for accessing the Airflow web interface

  auth:
    jwtSecret: CHANGE_ME # REQUIRED: generate with "openssl rand -base64 48"
    jwtExpirationDelta: 3600 # JWT token expiration time in seconds
    jwtRefreshThreshold: 300 # Threshold before token expires to allow refresh (seconds)
    fernetKey: CHANGE_ME # REQUIRED: generate with "python3 -c 'from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())'"

  admin:
    username: admin
    password: CHANGE_ME # REQUIRED: change before deploying to production

  scheduler:
    dagDirListInterval: 10 # How often to check DAG folder (seconds)
    minFileProcessInterval: 10 # Minimum interval to process DAG files (seconds)

  celery:
    workerConcurrency: 1 # Number of tasks each worker can run concurrently

volumeset:
  airflow:
    capacity: 10 # initial capacity in GiB (minimum is 10)

# Firewall configuration
firewallConfig:
  inboundAllowCIDR:
    - 0.0.0.0/0 # Restrict to specific IPs in production (e.g. - 203.0.113.0/24)

# Git-sync configuration for DAG delivery
gitSync:
  enabled: false
  repo: "" # Git repository URL (e.g. https://github.com/org/dags)
  branch: main # Branch to sync
  period: 60s # How often to sync
  subPath: "" # Optional subfolder within the repo containing DAGs
  auth:
    token: "" # Personal access token for private repos (leave empty for public repos)

# KEDA (Kubernetes Event-driven Autoscaling) configuration
# NOT SUPPORTED in gcp/us-central1
keda:
  enabled: true # Enable or disable KEDA autoscaling
  minScale: 1 # Minimum number of Celery workers
  maxScale: 3 # Maximum number of Celery workers
  scaleToZeroDelay: 300 # Time before scaling to zero (seconds)
  listLength: 3 # Queue length threshold to trigger scaling
  cooldownPeriod: 1 # Cooldown between scaling events (seconds)
  initialCooldownPeriod: 1 # Cooldown after startup before scaling (seconds)
  pollingInterval: 4 # Interval at which KEDA queries metrics (seconds)
```

### GVC

* `gvc.name` — The name of the GVC. Must be unique per deployment.
* `gvc.locations` — List of cloud locations to deploy to (e.g., `aws-eu-central-1`).

### PostgreSQL

* `postgres.image` — PostgreSQL Docker image.
* `postgres.resources` — CPU and memory bounds for the PostgreSQL workload (`minCpu`, `maxCpu`, `minMemory`, `maxMemory`).
* `postgres.config.username` / `postgres.config.password` — Database credentials. **Change the default password before deploying to production.**
* `postgres.config.database` — Name of the Airflow metadata database (default: `airflow`).
* `postgres.volumeset.capacity` — Persistent storage for PostgreSQL data (GiB, minimum 10).

### Redis

* `redis.image` — Redis Docker image.
* `redis.resources` — CPU and memory allocated to Redis.
* `redis.volumeset.capacity` — Persistent storage for Redis data (GiB, minimum 10).

### Airflow Webserver and Workers

* `airflow.webserver.image` / `airflow.celeryWorker.image` — Docker images for the webserver and Celery workers.
* `airflow.webserver.resources` / `airflow.celeryWorker.resources` — CPU and memory per component.
* `airflow.webPort` — Port the Airflow web UI listens on (default `8080`).

### Authentication

Airflow 3.x requires three security credentials, all of which **must be changed before deploying to production**:

* `airflow.auth.jwtSecret` — Secret key used to sign JWT tokens for API authentication. Generate a secure value with:
  ```bash theme={null}
  openssl rand -base64 48
  ```
* `airflow.auth.fernetKey` — Key used to encrypt stored connections and variables. Generate with:
  ```bash theme={null}
  python3 -c 'from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())'
  ```
* `airflow.auth.jwtExpirationDelta` — Token lifetime in seconds (default `3600`).
* `airflow.auth.jwtRefreshThreshold` — Seconds before expiry at which a token refresh is allowed (default `300`).

### Admin Account

* `airflow.admin.username` — Username for the initial Airflow admin account (default: `admin`).
* `airflow.admin.password` — Password for the initial admin account. **Change before deploying to production.**

The admin user is created on first startup using Airflow's `SimpleAuthManager`. Credentials are written to a password file on the shared volume and re-applied on every container restart, so the password always reflects the current value in `values.yaml`.

<Note>
  `SimpleAuthManager` is the default auth manager in Airflow 3.x and is suitable for development and internal deployments. For production deployments requiring SSO or LDAP, consider integrating an external auth provider via OAuth/OIDC.
</Note>

### Scheduler

* `airflow.scheduler.dagDirListInterval` — How often the scheduler scans the DAG folder for new or modified files (seconds).
* `airflow.scheduler.minFileProcessInterval` — Minimum interval between processing the same DAG file (seconds).

### Celery

* `airflow.celery.workerConcurrency` — Number of tasks a single Celery worker can execute concurrently.

### Storage

* `volumeset.airflow.capacity` — Persistent storage for the Airflow home directory shared across workloads (GiB, minimum 10).

PostgreSQL and Redis storage are configured under their respective sections (`postgres.volumeset.capacity` and `redis.volumeset.capacity`).

<Note>
  The Airflow volume uses a shared (NFS-style) filesystem, allowing both the webserver and Celery workers to read DAGs and write logs to the same volume.
</Note>

### Firewall

* `firewallConfig.inboundAllowCIDR` — List of CIDR ranges allowed to reach the Airflow webserver. Defaults to `0.0.0.0/0` (public). **Restrict to specific IP ranges in production.**

### Git-Sync

Git-sync runs as a sidecar container on the webserver and Celery worker workloads, continuously pulling DAGs from a Git repository into the shared Airflow volume. This is the recommended approach for managing DAGs in production.

| Property             | Description                                                                   |
| -------------------- | ----------------------------------------------------------------------------- |
| `gitSync.enabled`    | Enable or disable the git-sync sidecar                                        |
| `gitSync.repo`       | Git repository URL (e.g. `https://github.com/org/dags`)                       |
| `gitSync.branch`     | Branch to sync (default: `main`)                                              |
| `gitSync.period`     | Sync interval (default: `60s`)                                                |
| `gitSync.subPath`    | Optional subfolder within the repo containing DAG files                       |
| `gitSync.auth.token` | Personal access token for private repositories (leave empty for public repos) |

When git-sync is disabled, DAGs can be placed manually in the `/opt/airflow/dags` directory on the Airflow volume.

### KEDA Autoscaling

KEDA scales Celery workers automatically based on the Redis queue length.

<Note>
  KEDA is not supported in `gcp/us-central1`.
</Note>

| Property                     | Description                                                |
| ---------------------------- | ---------------------------------------------------------- |
| `keda.enabled`               | Enable or disable KEDA autoscaling                         |
| `keda.minScale`              | Minimum number of Celery workers                           |
| `keda.maxScale`              | Maximum number of Celery workers                           |
| `keda.scaleToZeroDelay`      | Seconds of inactivity before scaling to zero               |
| `keda.listLength`            | Redis queue length that triggers a scale-up                |
| `keda.cooldownPeriod`        | Seconds to wait between scaling events                     |
| `keda.initialCooldownPeriod` | Seconds after startup before autoscaling activates         |
| `keda.pollingInterval`       | Interval at which KEDA queries Redis for metrics (seconds) |

### Connecting to Airflow

Once deployed, the Airflow web UI is available at the workload's canonical endpoint:

```text theme={null}
https://<gvc-name>-airflow-webserver.<gvc-name>.cpln.app
```

Log in with the `airflow.admin.username` and `airflow.admin.password` set in `values.yaml`.

<Note>
  This template creates a GVC with a default name defined in the values file. If you plan to deploy multiple instances, you **must assign a unique GVC name** for each deployment.
</Note>

### API Access

Airflow 3.x uses JWT-based authentication for API access. To obtain a token:

```bash theme={null}
curl -X POST https://<your-airflow-url>/auth/token \
  -H "Content-Type: application/json" \
  -d '{"username": "admin", "password": "your-password"}'
```

Use the returned token for subsequent API requests:

```bash theme={null}
curl https://<your-airflow-url>/api/v2/dags \
  -H "Authorization: Bearer <token>"
```

## Production Considerations

* **Change all `CHANGE_ME` values** before deploying — `jwtSecret`, `fernetKey`, `admin.password`, and `postgres.config.password` are all required.
* **Restrict `firewallConfig.inboundAllowCIDR`** to trusted IP ranges to limit access to the Airflow UI.
* **Enable git-sync** for reliable, version-controlled DAG delivery.
* **Auth**: `SimpleAuthManager` is not recommended for deployments requiring enterprise SSO. Evaluate an OAuth/OIDC integration for those use cases.

## External References

<CardGroup cols={2}>
  <Card title="Apache Airflow Documentation" icon="book" href="https://airflow.apache.org/docs/">
    Official Apache Airflow documentation
  </Card>

  <Card title="CeleryExecutor" icon="layer-group" href="https://airflow.apache.org/docs/apache-airflow-providers-celery/stable/celery_executor.html">
    Learn about the CeleryExecutor and distributed task execution
  </Card>

  <Card title="KEDA Documentation" icon="chart-line" href="https://keda.sh/docs/">
    Kubernetes Event-driven Autoscaling documentation
  </Card>

  <Card title="Redis Documentation" icon="database" href="https://redis.io/docs/latest/">
    Official Redis documentation
  </Card>

  <Card title="git-sync" icon="github" href="https://github.com/kubernetes/git-sync">
    git-sync sidecar documentation
  </Card>

  <Card title="Airflow Template" icon="github" href="https://github.com/controlplane-com/templates/tree/main/airflow">
    View the source files, default values, and chart definition
  </Card>
</CardGroup>