> ## Documentation Index
> Fetch the complete documentation index at: https://docs.controlplane.com/llms.txt
> Use this file to discover all available pages before exploring further.

# CDC Pipeline

> Deploy a complete Change Data Capture pipeline on Control Plane in a single install. Bundles PostgreSQL HA (Patroni + etcd + HAProxy), Apache Kafka (KRaft), and Debezium Server with auto-coordinated credentials, WAL settings, and internal DNS.

## Overview

The CDC Pipeline is a meta-template that deploys a complete, pre-wired Change Data Capture pipeline in a single install. It bundles three production-ready templates — PostgreSQL HA, Apache Kafka, and Debezium Server — and automatically coordinates the credentials, WAL settings, and internal hostnames that would otherwise require manual configuration between them.

### Architecture

* **PostgreSQL HA** (Patroni + etcd + HAProxy) — Source database with logical replication enabled (`wal_level = logical`). HAProxy routes writes to the current primary; replicas are addressable directly.
* **Kafka** (KRaft, no ZooKeeper) — Event streaming platform. Debezium publishes change events here; consumers read from it.
* **Debezium Server** — CDC connector that tails the PostgreSQL WAL, converts row-level changes to structured events, and produces them to Kafka via SASL\_PLAINTEXT.

Internal DNS names are automatically derived from the release name at install time:

| Service                  | Internal Hostname                                        |
| ------------------------ | -------------------------------------------------------- |
| PostgreSQL (via HAProxy) | `{release-name}-postgres-ha-proxy.{gvc}.cpln.local:5432` |
| Kafka                    | `{release-name}-cluster.{gvc}.cpln.local:9092`           |
| Debezium                 | `{release-name}-debezium.{gvc}.cpln.local`               |

### What Gets Created

Everything from each bundled template:

* **From PostgreSQL HA**: Stateful PostgreSQL Workload (3 replicas), etcd cluster, HAProxy workload, Volume Sets, secrets, identity and policy.
* **From Kafka**: Stateful Kafka broker workloads (KRaft, 3 replicas), Kafka Exporter, JMX Exporter, Kafbat UI, Volume Sets, secrets, identity and policy.
* **From Debezium Server**: Standard Debezium Server Workload, Volume Set (file offset storage), secrets, identity and policy.

<Note>
  This template does not create a GVC. You must deploy it into an existing GVC.
</Note>

## Prerequisites

This template has no external prerequisites — PostgreSQL, Kafka, and Debezium are all deployed together and auto-configured. To install, follow the instructions for your preferred method:

<CardGroup cols={2}>
  <Card title="UI" href="/template-catalog/install-manage/ui" icon="laptop">
    Browse, install, and manage templates visually
  </Card>

  <Card title="CLI" href="/template-catalog/install-manage/cli" icon="terminal">
    Manage templates from your terminal
  </Card>

  <Card title="Terraform" href="/template-catalog/install-manage/terraform" icon={<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 128 128"><g fill-rule="evenodd"><path d="M77.941 44.5v36.836L46.324 62.918V26.082zm0 0" fill="#5c4ee5"/><path d="M81.41 81.336l31.633-18.418V26.082L81.41 44.5zm0 0" fill="#4040b2"/><path d="M11.242 42.36L42.86 60.776V23.941L11.242 5.523zm0 0M77.941 85.375L46.324 66.957v36.82l31.617 18.418zm0 0" fill="#5c4ee5"/></g></svg>}>
    Declare templates in your Terraform configurations
  </Card>

  <Card
    title="Pulumi"
    href="/template-catalog/install-manage/pulumi"
    icon={<svg xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 24 24" id="Pulumi-Icon--Streamline-Svg-Logos" height="24" width="24">
    <desc>
        Pulumi Icon Streamline Icon: https://streamlinehq.com
    </desc>
    <path fill="#f26e7e" d="M4.683025 13.3318c0.869125 -0.5018 0.870575 -2.1264 0.003225 -3.62865s-2.27504 -2.313275 -3.1441725 -1.811475C0.672945 8.3935 0.6715 10.0181 1.53885 11.52035c0.86735 1.502275 2.27505 2.313275 3.144175 1.81145Zm0.0052 3.2167c0.86735 1.502275 0.865925 3.126875 -0.003225 3.628675 -0.86915 0.5018 -2.2768275 -0.309225 -3.144175 -1.81145 -0.8673525 -1.50225 -0.8659075 -3.126875 0.003225 -3.628675 0.8691325 -0.5018 2.276825 0.309225 3.144175 1.81145Zm5.922875 3.4243c0.86735 1.50225 0.8659 3.126775 -0.003225 3.62875 -0.869125 0.501775 -2.27685 -0.309325 -3.1442 -1.81155 -0.867325 -1.50225 -0.865875 -3.12685 0.00325 -3.628675 0.869125 -0.5018 2.276825 0.309225 3.144175 1.811475Zm-0.001925 -6.845275c0.86735 1.50225 0.8659 3.12685 -0.003225 3.628675 -0.869125 0.5018 -2.276825 -0.309225 -3.144175 -1.811475 -0.86735 -1.50225 -0.8659 -3.12685 0.003225 -3.62865 0.869125 -0.501825 2.276825 0.3092 3.144175 1.81145Z" stroke-width="0.25"></path>
    <path fill="#8a3391" d="M22.45775 11.524125c0.86725 -1.502225 0.865925 -3.12685 -0.003225 -3.62865 -0.869125 -0.501825 -2.276825 0.3092 -3.144175 1.811475 -0.86735 1.50225 -0.8659 3.126825 0.003225 3.62865 0.869125 0.501825 2.276825 -0.3092 3.144175 -1.811475Zm0.000175 3.2151c0.869075 0.5018 0.870625 2.1264 0.003225 3.62865 -0.86735 1.50225 -2.27505 2.313275 -3.144175 1.81145 -0.869125 -0.5018 -0.870575 -2.126425 -0.003225 -3.62865 0.86735 -1.50225 2.27505 -2.313275 3.144175 -1.81145ZM16.536225 18.157875c0.86915 0.501825 0.8706 2.126425 0.00325 3.628675 -0.86735 1.502125 -2.275075 2.313225 -3.1442 1.81145 -0.869125 -0.50175 -0.870575 -2.126425 -0.003225 -3.62865 0.867375 -1.502275 2.27505 -2.3133 3.144175 -1.811475Zm-0.003325 -6.843775c0.869125 0.5018 0.870575 2.126425 0.003225 3.628675s-2.27505 2.313275 -3.1442 1.811475c-0.869125 -0.501825 -0.870575 -2.126425 -0.003225 -3.628675 0.86735 -1.502275 2.27505 -2.313275 3.1442 -1.811475Z" stroke-width="0.25"></path>
    <path fill="#f7bf2a" d="M15.138225 2.06721c0 1.003615 -1.40625 1.817215 -3.14095 1.817215 -1.7347 0 -3.14095 -0.8136 -3.14095 -1.817215C8.856325 1.06359 10.262575 0.25 11.997275 0.25c1.7347 0 3.14095 0.81359 3.14095 1.81721ZM9.2166 5.482375c0 1.003625 -1.40625 1.8172 -3.14095 1.8172 -1.7347 0 -3.14095 -0.813575 -3.14095 -1.8172s1.40625 -1.817225 3.14095 -1.817225c1.7347 0 3.14095 0.8136 3.14095 1.817225Zm8.71005 1.8172c1.7347 0 3.14095 -0.813575 3.14095 -1.8172s-1.40625 -1.817225 -3.14095 -1.817225c-1.7347 0 -3.14095 0.8136 -3.14095 1.817225s1.40625 1.8172 3.14095 1.8172Zm-2.788425 1.605625c0 1.003625 -1.40625 1.8172 -3.14095 1.8172 -1.7347 0 -3.14095 -0.813575 -3.14095 -1.8172 0 -1.0036 1.40625 -1.8172 3.14095 -1.8172 1.7347 0 3.14095 0.8136 3.14095 1.8172Z" stroke-width="0.25"></path>
    </svg>}
  >
    Declare templates in your Pulumi programs
  </Card>
</CardGroup>

## Configuration

The first thing to do after installing is **replace all `changeme-*` placeholder values** with real secrets. The template validates at deploy time that shared credentials are consistent — mismatches are caught before any workload starts.

### Shared Credentials

These values are used by multiple components and must be set once in a coordinated way. The default `values.yaml` pre-wires them:

**PostgreSQL ↔ Debezium:**

| Setting       | PostgreSQL path                               | Debezium path                              |
| ------------- | --------------------------------------------- | ------------------------------------------ |
| Database name | `postgres-highly-available.postgres.database` | `debezium-server.source.database.name`     |
| Username      | `postgres-highly-available.postgres.username` | `debezium-server.source.database.user`     |
| Password      | `postgres-highly-available.postgres.password` | `debezium-server.source.database.password` |

**Kafka ↔ Debezium:**

| Setting       | Kafka path                                    | Debezium path                             |
| ------------- | --------------------------------------------- | ----------------------------------------- |
| SASL username | `kafka.kafka.listeners.client.sasl.users`     | `debezium-server.sink.kafka.saslUsername` |
| SASL password | `kafka.kafka.listeners.client.sasl.passwords` | `debezium-server.sink.kafka.saslPassword` |

### Deploy-time Validation

The template validates at install time that:

* `postgres-highly-available.postgres.walLevel` is set to `logical` (required for CDC)
* Database credentials match between the PostgreSQL and Debezium blocks
* Debezium's Kafka SASL username appears in Kafka's configured user list

If any of these are mismatched, the install fails with a descriptive error before any resources are created.

### Key Configuration Sections

#### What Tables to Capture

Set `debezium-server.source.tableIncludeList` to a comma-separated list of `schema.table` names:

```yaml theme={null}
debezium-server:
  source:
    tableIncludeList: "public.users,public.orders,public.products"
```

Leave empty to capture all tables.

#### Debezium Heartbeat (Recommended)

The default configuration enables Debezium heartbeats every 5 seconds to keep the replication slot active and prevent WAL accumulation during low-traffic periods.

After the pipeline is running, create the heartbeat table in PostgreSQL:

```sql theme={null}
CREATE TABLE IF NOT EXISTS debezium_heartbeat (id INT PRIMARY KEY, ts TIMESTAMPTZ);
INSERT INTO debezium_heartbeat VALUES (1, now());
```

#### Resource Sizing

Adjust resources per component based on your workload:

```yaml theme={null}
postgres-highly-available:
  resources:
    minCpu: 500m
    minMemory: 1Gi
    maxCpu: 1
    maxMemory: 2Gi

kafka:
  kafka:
    cpu: 1000m
    memory: 2000Mi

debezium-server:
  resources:
    cpu: 500m
    memory: 512Mi
```

#### Component Versions

| Component       | Template Version | Software Version           |
| --------------- | ---------------- | -------------------------- |
| PostgreSQL HA   | 2.2.0            | PostgreSQL 17 (Patroni)    |
| Kafka           | 4.0.0            | Apache Kafka 3.9.1 (KRaft) |
| Debezium Server | 1.1.0            | Debezium 3.0               |

### Using External PostgreSQL or Kafka

To replace the bundled PostgreSQL or Kafka with an external instance, override the hostname/bootstrap servers explicitly. The template will still deploy the other bundled components:

```yaml theme={null}
debezium-server:
  source:
    database:
      hostname: "my-external-postgres.example.com"

  sink:
    kafka:
      bootstrapServers: "my-external-kafka.example.com:9092"
```

When using an external PostgreSQL, ensure logical replication is already enabled (`wal_level = logical`) before installing.

## Connecting Consumers to Kafka

Applications consuming change events connect to Kafka using SASL\_PLAINTEXT. The internal Kafka bootstrap address:

```
{release-name}-cluster.{gvc}.cpln.local:9092
```

Example Kafka client properties:

```properties theme={null}
bootstrap.servers={release-name}-cluster.{gvc}.cpln.local:9092
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
  username="debezium" \
  password="your-kafka-debezium-password";
```

Change events are published to topics named `{serverName}.{schema}.{table}` — e.g. `dbserver1.public.users`.

## Kafbat UI

The Kafbat UI (Kafka web dashboard) is included and enabled by default. Access it via `cpln workload connect` or configure its `firewall.external_inboundAllowCIDR` to expose it externally.

## External References

<CardGroup cols={2}>
  <Card title="Debezium Server Template" href="/template-catalog/templates/debezium-server" icon="arrow-right-arrow-left">
    Deploy Debezium Server standalone against any PostgreSQL, MySQL, or other source
  </Card>

  <Card title="PostgreSQL Highly Available Template" href="/template-catalog/templates/postgres-highly-available" icon="database">
    The HA PostgreSQL template bundled in this pipeline
  </Card>

  <Card title="Kafka Template" href="/template-catalog/templates/kafka" icon="bezier-curve">
    The Kafka template bundled in this pipeline
  </Card>

  <Card title="Debezium Documentation" href="https://debezium.io/documentation/" icon="book">
    Full Debezium connector and server reference
  </Card>
</CardGroup>