Skip to main content

​
Overview

The CDC Pipeline is a meta-template that deploys a complete, pre-wired Change Data Capture pipeline in a single install. It bundles three production-ready templates — PostgreSQL HA, Apache Kafka, and Debezium Server — and automatically coordinates the credentials, WAL settings, and internal hostnames that would otherwise require manual configuration between them.

​
Architecture

  • PostgreSQL HA (Patroni + etcd + HAProxy) — Source database with logical replication enabled (wal_level = logical). HAProxy routes writes to the current primary; replicas are addressable directly.
  • Kafka (KRaft, no ZooKeeper) — Event streaming platform. Debezium publishes change events here; consumers read from it.
  • Debezium Server — CDC connector that tails the PostgreSQL WAL, converts row-level changes to structured events, and produces them to Kafka via SASL_PLAINTEXT.
Internal DNS names are automatically derived from the release name at install time:
ServiceInternal Hostname
PostgreSQL (via HAProxy){release-name}-postgres-ha-proxy.{gvc}.cpln.local:5432
Kafka{release-name}-cluster.{gvc}.cpln.local:9092
Debezium{release-name}-debezium.{gvc}.cpln.local

​
What Gets Created

Everything from each bundled template:
  • From PostgreSQL HA: Stateful PostgreSQL Workload (3 replicas), etcd cluster, HAProxy workload, Volume Sets, secrets, identity and policy.
  • From Kafka: Stateful Kafka broker workloads (KRaft, 3 replicas), Kafka Exporter, JMX Exporter, Kafbat UI, Volume Sets, secrets, identity and policy.
  • From Debezium Server: Standard Debezium Server Workload, Volume Set (file offset storage), secrets, identity and policy.
This template does not create a GVC. You must deploy it into an existing GVC.

​
Prerequisites

This template has no external prerequisites — PostgreSQL, Kafka, and Debezium are all deployed together and auto-configured. To install, follow the instructions for your preferred method:

UI

Browse, install, and manage templates visually

CLI

Manage templates from your terminal

Terraform

Declare templates in your Terraform configurations
Pulumi Icon Streamline Icon: https://streamlinehq.com

Pulumi

Declare templates in your Pulumi programs

​
Configuration

The first thing to do after installing is replace all changeme-* placeholder values with real secrets. The template validates at deploy time that shared credentials are consistent — mismatches are caught before any workload starts.

​
Shared Credentials

These values are used by multiple components and must be set once in a coordinated way. The default values.yaml pre-wires them: PostgreSQL ↔ Debezium:
SettingPostgreSQL pathDebezium path
Database namepostgres-highly-available.postgres.databasedebezium-server.source.database.name
Usernamepostgres-highly-available.postgres.usernamedebezium-server.source.database.user
Passwordpostgres-highly-available.postgres.passworddebezium-server.source.database.password
Kafka ↔ Debezium:
SettingKafka pathDebezium path
SASL usernamekafka.kafka.listeners.client.sasl.usersdebezium-server.sink.kafka.saslUsername
SASL passwordkafka.kafka.listeners.client.sasl.passwordsdebezium-server.sink.kafka.saslPassword

​
Deploy-time Validation

The template validates at install time that:
  • postgres-highly-available.postgres.walLevel is set to logical (required for CDC)
  • Database credentials match between the PostgreSQL and Debezium blocks
  • Debezium’s Kafka SASL username appears in Kafka’s configured user list
If any of these are mismatched, the install fails with a descriptive error before any resources are created.

​
Key Configuration Sections

​
What Tables to Capture

Set debezium-server.source.tableIncludeList to a comma-separated list of schema.table names: 
debezium-server:
  source:
    tableIncludeList: "public.users,public.orders,public.products"
Leave empty to capture all tables. The default configuration enables Debezium heartbeats every 5 seconds to keep the replication slot active and prevent WAL accumulation during low-traffic periods. After the pipeline is running, create the heartbeat table in PostgreSQL: 
CREATE TABLE IF NOT EXISTS debezium_heartbeat (id INT PRIMARY KEY, ts TIMESTAMPTZ);
INSERT INTO debezium_heartbeat VALUES (1, now());

​
Resource Sizing

Adjust resources per component based on your workload: 
postgres-highly-available:
  resources:
    minCpu: 500m
    minMemory: 1Gi
    maxCpu: 1
    maxMemory: 2Gi

kafka:
  kafka:
    cpu: 1000m
    memory: 2000Mi

debezium-server:
  resources:
    cpu: 500m
    memory: 512Mi

​
Component Versions

ComponentTemplate VersionSoftware Version
PostgreSQL HA2.2.0PostgreSQL 17 (Patroni)
Kafka4.0.0Apache Kafka 3.9.1 (KRaft)
Debezium Server1.1.0Debezium 3.0

​
Using External PostgreSQL or Kafka

To replace the bundled PostgreSQL or Kafka with an external instance, override the hostname/bootstrap servers explicitly. The template will still deploy the other bundled components: 
debezium-server:
  source:
    database:
      hostname: "my-external-postgres.example.com"

  sink:
    kafka:
      bootstrapServers: "my-external-kafka.example.com:9092"
When using an external PostgreSQL, ensure logical replication is already enabled (wal_level = logical) before installing.

​
Connecting Consumers to Kafka

Applications consuming change events connect to Kafka using SASL_PLAINTEXT. The internal Kafka bootstrap address: 
{release-name}-cluster.{gvc}.cpln.local:9092
Example Kafka client properties: 
bootstrap.servers={release-name}-cluster.{gvc}.cpln.local:9092
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
  username="debezium" \
  password="your-kafka-debezium-password";
Change events are published to topics named {serverName}.{schema}.{table} — e.g. dbserver1.public.users.

​
Kafbat UI

The Kafbat UI (Kafka web dashboard) is included and enabled by default. Access it via cpln workload connect or configure its firewall.external_inboundAllowCIDR to expose it externally.

​
External References

Debezium Server Template

Deploy Debezium Server standalone against any PostgreSQL, MySQL, or other source

PostgreSQL Highly Available Template

The HA PostgreSQL template bundled in this pipeline

Kafka Template

The Kafka template bundled in this pipeline

Debezium Documentation

Full Debezium connector and server reference