Kafka - Control Plane

Overview

Apache Kafka is a distributed event streaming platform for high-throughput, fault-tolerant publish-subscribe messaging. This template deploys a Kafka cluster in KRaft mode (no ZooKeeper): every broker is also a KRaft controller, the client listener authenticates with SASL/PLAIN, Kafka ACLs are enforced, and each broker keeps its logs on persistent volumes with daily snapshots. A health sidecar and two Prometheus exporters run beside every broker. Kafbat UI (a web console), a Confluent REST Proxy and a Kafka CLI client workload are on by default; Kafka Connect workers and a public listener for clients outside the GVC are opt-in.

This template deploys into an existing GVC — the one you pass with --gvc. It does not create, provision or manage a GVC, and uninstalling the release never deletes one.

What Gets Created

Resource	Name	Purpose	Optional
Workload	`RELEASE_NAME-cluster`	Kafka brokers (stateful, `kafka.replicas` replicas, each a KRaft controller and broker). Containers: `kafka`, `kafka-orchestrator` (readiness and metrics), `kafka-exporter`, `jmx-exporter`	Set `kafka_orchestrator`, `kafka_exporter` or `jmx_exporter` to `null` to drop that sidecar
Volume Set	`RELEASE_NAME-logs-0`, `RELEASE_NAME-logs-1`	One per path in `kafka.logDirs`; ext4 SSD, autoscaling, daily snapshots	`kafka.volumes.logs.externalVolumeSets` mounts existing volume sets instead
Secret	`RELEASE_NAME-controller-configuration`	Broker `server.properties` (listeners, SASL, ACL, replication defaults)	—
Secret	`RELEASE_NAME-init`	Broker startup script	—
Secret	`RELEASE_NAME-secrets`	KRaft cluster ID, controller, inter-broker and listener admin passwords	—
Secret	`RELEASE_NAME-jmx-exporter-conf`	JMX exporter rules	with `jmx_exporter`
Identity and Policy	`RELEASE_NAME`	`reveal` on the broker secrets	—
Workload	`RELEASE_NAME-kafbat-ui`	Kafbat UI web console (standard), with an identity and policy of the same name reading your configuration secret	`kafbat_ui.enabled: true` (default)
Workload	`RELEASE_NAME-rest-proxy`	Confluent REST Proxy (standard) on port 8082, with identity `RELEASE_NAME-rest-proxy-identity`, policy `RELEASE_NAME-rest-proxy-policy` and secrets `RELEASE_NAME-rest-properties`, `RELEASE_NAME-rest-jaas-conf` (plus `RELEASE_NAME-rest-password-properties` when BASIC-auth users are set)	`kafka_rest_proxy.enabled: true` (default)
Workload	`RELEASE_NAME-client`	Kafka CLI tools container for testing from inside the GVC	`kafka_client` block (default on; set to `null` to omit)
Workload	`RELEASE_NAME-connect-` followed by the connector `name`	Kafka Connect worker (stateful). Containers: `kafka-connect` and `plugins-downloader`. Comes with a plugins volume set, identity and policy of the same name, and secrets with the suffixes `-props`, `-init` and `-download`	`kafka_connectors` list
Domain	the hostname you configure	TLS entry point for the public listener, Kafbat UI or REST Proxy	`kafka.listeners.<listener>.directReplicaRouting.publicAddress`, `kafbat_ui.domain`, `kafka_rest_proxy.domain`

Prerequisites

Create the Kafbat UI configuration secret

Kafbat UI is on by default and reads its whole configuration from an opaque secret mounted as a file. The secret must exist before you install. Replace ADMIN_PASSWORD with the value you will pass as kafka.listeners.client.sasl.admin.password, and UI_PASSWORD with the login you want for the console. The full file format is in the Kafbat configuration reference.

cat > kafbat-config.yaml <<'EOF'
kafka:
  clusters:
    - name: kafka
      bootstrapServers: RELEASE_NAME-cluster.GVC_NAME.cpln.local:9092
      properties:
        security.protocol: SASL_PLAINTEXT
        sasl.mechanism: PLAIN
        sasl.jaas.config: "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"admin\" password=\"ADMIN_PASSWORD\";"
auth:
  type: LOGIN_FORM
spring:
  security:
    user:
      name: admin
      password: UI_PASSWORD
EOF
cpln secret create-opaque --name kafka-kafbat-ui-config --encoding plain -f kafbat-config.yaml

kafka-kafbat-ui-config is the default value of kafbat_ui.configuration_secret; if you pick another name, set that value to match. If you do not want the console, install with --set kafbat_ui.enabled=false instead and skip this step.

If the secret is missing at install time the install still succeeds, cpln logs returns nothing for RELEASE_NAME-kafbat-ui, and the workload never becomes ready. The only place the missing secret is named is status.versions[].message:

cpln workload get-deployments RELEASE_NAME-kafbat-ui --gvc GVC_NAME -o yaml

Create the secret and the deployment recovers on its own after a few minutes, or force it with cpln workload force-redeployment RELEASE_NAME-kafbat-ui --gvc GVC_NAME.

Choose the cluster credentials

The KRaft cluster ID and the SASL passwords are template values, set at install time. Generate a cluster ID with the Kafka tooling (the same image the template runs):

docker run --rm apache/kafka:3.9.1 /opt/kafka/bin/kafka-storage.sh random-uuid

Pick strong values for kafka.secrets.inter_broker_password, kafka.secrets.controller_password, kafka.listeners.client.sasl.admin.password and kafka.listeners.client.sasl.passwords. The your-… defaults in values.yaml are placeholders that work as literal passwords, so never leave them in place. The cluster ID is written into the log volumes at first start and cannot be changed afterwards without a new cluster.

Public listener (optional)

Exposing brokers to clients outside the GVC needs a domain you control and a dedicated load balancer enabled on the GVC. Follow the Configure Domain guide first, then see Public Listener below.

Installation

Install into an existing GVC, supplying the credentials from the prerequisites:

cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/kafka \
  --version 4.3.0 --gvc GVC_NAME \
  --set kafka.secrets.kraft_cluster_id=KRAFT_CLUSTER_ID \
  --set kafka.secrets.inter_broker_password=INTER_BROKER_PASSWORD \
  --set kafka.secrets.controller_password=CONTROLLER_PASSWORD \
  --set kafka.listeners.client.sasl.admin.password=ADMIN_PASSWORD \
  --set kafka.listeners.client.sasl.passwords=USER_PASSWORD

The REST Proxy ships pointing at a bootstrap address and admin password that only match a release named kafka-dev with the default password. Either put kafka_rest_proxy.properties.bootstrap.servers, kafka_rest_proxy.jaas_conf and kafka_rest_proxy.password_properties in a values file (-f values.yaml) as shown under Kafka REST Proxy, or add --set kafka_rest_proxy.enabled=false.

UI

Browse, install, and manage templates visually

CLI

Manage templates from your terminal

Terraform

Declare templates in your Terraform configurations

Pulumi

Declare templates in your Pulumi programs

Configuration

Each block below shows the shipped defaults for one area of values.yaml. Only the keys you change need to go in your values file.

Kafka Cluster

kafka:
  name: cluster # workload is RELEASE_NAME-cluster
  image: apache/kafka:3.9.1 # Apache images only; Bitnami images are rejected at render time
  suspend: false
  deletionProtection: false
  replicas: 3 # 1, 3, 4 or 5 — must not be 2 and cannot exceed 5
  minReadySeconds: 0
  debug: false
  multiZone: false # true spreads brokers across availability zones; enable multi-zone on the GVC's dedicated load balancer too
  rackAwareness:
    enabled: false # AWS-only: brokers advertise their zone as broker.rack so consumers that set client.rack read from a same-zone replica
  logDirs: /opt/kafka/logs-0,/opt/kafka/logs-1 # one volume set per path
  env: [] # extra environment variables for the kafka container
  terminationGracePeriodSeconds: 600 # time allowed for a clean broker shutdown; raise for very large brokers
  cpu: 1000m
  memory: 2000Mi # the JVM heap defaults to 60% of this
  minCpu: 250m
  minMemory: 2000Mi
  # overrideHeapOpts: "-Xmx1024m -Xms1024m" # replace the derived heap settings

Storage

kafka:
  volumes:
    logs:
      initialCapacity: 10 # In GB
      performanceClass: general-purpose-ssd # general-purpose-ssd / high-throughput-ssd (Min 1000GB)
      fileSystemType: ext4 # ext4 / xfs
      externalVolumeSets: [] # mount existing volume sets instead of creating them: one entry per logDirs path, in order; "" keeps that one chart-managed
      snapshots:
        createFinalSnapshot: true
        retentionDuration: 7d
        schedule: 0 0 * * * # UTC
      autoscaling:
        maxCapacity: 1000 # In GB
        minFreePercentage: 20
        scalingFactor: 1.2
      # customEncryption: # encrypt the broker volumes with your own AWS KMS key; only possible when the volume sets are first created
      #   enabled: true
      #   region: aws-us-east-2
      #   keyId: arn:aws:kms:us-east-2:1234567890:key/d411f35a-1d31-4515-9934-4f193e042d80

With customEncryption enabled, finish the setup after installing: open each RELEASE_NAME-logs-N volume set in the console, go to its spec and follow the AWS Custom Encryption Instructions shown there. Existing volumes cannot be re-encrypted.

Firewall

kafka:
  firewall:
    internal_inboundAllowType: "same-gvc" # same-gvc (recommended) / same-org
    # external_inboundAllowCIDR: 0.0.0.0/0 # comma-separated list; leave commented to keep the brokers off the internet
    # inboundAllowWorkload: # additional workloads allowed in, with same-gvc or workload-list
    #   - //gvc/main-kafka/workload/main-kafka-kafbat-ui
    #   - //gvc/client-gvc/workload/client
    # external_outboundAllowCIDR: "111.222.333.444/16,111.222.444.333/32" # comma-separated list

Listeners

Every entry under kafka.listeners becomes a broker listener. The chart ships one, client; a second entry is how you add external access. Ports 9091, 9093 and 9094 are reserved for the controller and inter-broker listeners.

Client Listener

kafka:
  listeners:
    client:
      protocol: SASL_PLAINTEXT # PLAINTEXT or SASL_PLAINTEXT
      name: CLIENT
      containerPort: 9092 # ignored when the listener has a publicAddress (ports 3000-3004 are used instead)
      sasl:
        admin:
          username: admin # also an ACL super user; the sidecars and exporters authenticate as this user
          password: "your-admin-password" # change before installing
        users: "user" # comma-separated list of additional users
        passwords: "your-user-password" # comma-separated, one per user, same order

Public Listener

Add a second listener to reach the brokers from outside the GVC. TLS is terminated at the load balancer, so external clients connect with security.protocol=SASL_SSL. Only one listener may carry a public address. Direct replica routing (recommended). One domain with a DNS01 certificate challenge; each broker is advertised at <replica>-<location>.<publicAddress>:<containerPort> (for release kafka in aws-us-east-1: kafka-cluster-0-aws-us-east-1.kafka.example.com:9095, kafka-cluster-1-aws-us-east-1.kafka.example.com:9095, …). Create a CNAME for each broker hostname plus the _acme-challenge record as the domain reference describes.

# goes under kafka.listeners, beside the shipped client listener
public:
  protocol: SASL_PLAINTEXT # external clients use SASL_SSL
  name: PUBLIC
  directReplicaRouting:
    enabled: true
    containerPort: 9095 # not 9091, 9093 or 9094
    publicAddress: kafka.example.com # a domain you own; the GVC needs a dedicated load balancer
  sasl:
    users: "public-user"
    passwords: "your-public-user-password"

Multi-port routing (legacy). One port per broker starting at 3000 on a single hostname (kafka.example.com:3000, :3001, :3002). Do not set containerPort here. Not recommended for multi-zone clusters, because traffic is not routed zone-aware.

# goes under kafka.listeners, beside the shipped client listener
public:
  protocol: SASL_PLAINTEXT
  name: PUBLIC
  publicAddress: kafka.example.com # ports 3000, 3001, … are created automatically, one per broker
  sasl:
    users: "public-user"
    passwords: "your-public-user-password"

ACL and Cluster Secrets

kafka:
  acl:
    superUsers: "User:admin" # semicolon-separated, e.g. User:admin;User:connectors
    allowEveryoneIfNoAclFound: false # true allows any operation on a resource that has no ACL
  secrets:
    kraft_cluster_id: your-kraft-cluster-id # kafka-storage.sh random-uuid, e.g. bkdDtS1Rsf536si7BGM0JY; fixed for the life of the cluster
    inter_broker_password: your-inter-broker-password # change before installing
    controller_password: your-controller-password # change before installing

Extra Configurations

Keys under kafka.extra_configurations are appended to server.properties and override the chart’s defaults. The chart derives default.replication.factor as the smaller of 3 and kafka.replicas, and min.insync.replicas as one less than that; set them here only to override.

kafka:
  extra_configurations:
    auto.create.topics.enable: true
    log.retention.hours: 168 # 7 days

Health Sidecar and Exporters

All three run as sidecar containers on RELEASE_NAME-cluster. The health sidecar gates broker readiness on real cluster health (broker registration, controller election, under-replicated partitions, log directories); the exporters publish Prometheus metrics that the platform scrapes automatically.

kafka_orchestrator: # readiness and metrics sidecar; set the whole block to null to remove it
  image: ghcr.io/controlplane-com/kafka-orchestrator:v0.1.0
  cpu: 100m
  memory: 128Mi
  minCpu: 50m
  minMemory: 64Mi
  listener: client # the kafka.listeners entry whose port and SASL admin user the health checks use
  logLevel: info # debug / info / warn / error
  checkTimeout: 10s # per-check timeout
  env: [] # e.g. BROKER_ID / BOOTSTRAP_SERVERS overrides for non-standard setups

kafka_exporter: # consumer-group lag and topic metrics; set to null to remove it
  name: exporter
  image: danielqsj/kafka-exporter:v1.9.0
  debug: false
  cpu: 50m
  memory: 128Mi
  listener: client
  env: []
  dropMetrics: [] # regex list, e.g. ["kafka_consumergroup.*", "^kafka_topic_partition_current_offset"]

jmx_exporter: # broker JVM and request metrics over JMX; set to null to remove it
  name: jmx-exporter
  image: ghcr.io/controlplane-com/bitnami/jmx-exporter
  kafkaJmxPort: 5557 # must match the port in config.jmxUrl
  exporterPort: 5556
  debug: false
  cpu: 250m
  memory: 256Mi
  minCpu: 80m
  minMemory: 125Mi
  listener: client
  dropMetrics: []
  # config: the JMX exporter rules file (jmxUrl, whitelistObjectNames, rules) — keep the chart defaults

Kafbat UI

kafbat_ui:
  enabled: true
  deletionProtection: false
  name: kafbat-ui # workload is RELEASE_NAME-kafbat-ui
  image: ghcr.io/kafbat/kafka-ui
  cpu: 300m
  memory: 1000Mi
  minCpu: 100m
  minMemory: 400Mi
  replicas: 1
  timeoutSeconds: 30
  configuration_secret: kafka-kafbat-ui-config # opaque secret you create before installing (see Prerequisites)
  # domain: kafbat-ui.example.com # custom domain for the console; create its DNS records first
  firewall:
    # internal_inboundAllowType: "same-gvc" # same-org / same-gvc
    external_inboundAllowCIDR: "0.0.0.0/0" # the console is reachable from the internet by default
    external_outboundAllowCIDR: "0.0.0.0/0"

Kafka REST Proxy

The proxy connects to the brokers as the KafkaClient principal in jaas_conf, and authenticates HTTP callers with BASIC auth against the users in password_properties. Both must be set for your release: bootstrap.servers must name your cluster workload and the KafkaClient password must equal kafka.listeners.client.sasl.admin.password. Without password_properties the BASIC realm has no users and every request is refused.

kafka_rest_proxy:
  enabled: true
  deletionProtection: false
  name: rest-proxy # workload is RELEASE_NAME-rest-proxy
  image: confluentinc/cp-kafka-rest:latest
  cpu: 500m
  memory: 1000Mi
  capacityAI:
    enabled: true
    minCpu: 125m # only applied when capacityAI is enabled
    minMemory: 200Mi # only applied when capacityAI is enabled
  replicas: 1
  timeoutSeconds: 15
  # domain: kafka-rest.example.com # custom domain for the proxy
  firewall:
    # internal_inboundAllowType: "same-gvc" # same-org / same-gvc
    external_inboundAllowCIDR: 0.0.0.0/0 # the proxy is reachable from the internet by default
    external_outboundAllowCIDR: "0.0.0.0/0"
  properties:
    bootstrap.servers: SASL_PLAINTEXT://kafka-dev-cluster:9092 # set to SASL_PLAINTEXT://RELEASE_NAME-cluster.GVC_NAME.cpln.local:9092
    resource.extension: ALL
    api.v3.enable: true
    api.v2.enable: true
    client.sasl.mechanism: PLAIN
    api.compatibility.mode: BOTH
    authentication.realm: KafkaRest
    authentication.method: BASIC
    authentication.roles: user
    client.security.protocol: SASL_PLAINTEXT
    # listeners and log4j.opts: keep the chart defaults; the proxy serves on port 8082
  # KafkaClient: the broker credentials (the client listener's admin user). KafkaRest: the HTTP BASIC-auth realm.
  jaas_conf:
    KafkaClient {
    org.apache.kafka.common.security.plain.PlainLoginModule required
    username="admin"
    password="your-admin-password";
    };
    KafkaRest {
      org.eclipse.jetty.jaas.spi.PropertyFileLoginModule required
      debug="true"
      file="/etc/kafka-rest/password.properties";
    };
  # password_properties: # HTTP BASIC-auth users, one per line as name: password,role (role must match authentication.roles)
  #   user: your-user-password,user

Kafka Connect

kafka_connectors is a list; each entry deploys a Kafka Connect worker cluster named RELEASE_NAME-connect- plus the entry’s name, with a volume set for plugins and a plugins-downloader container that fetches plugin artifacts into plugins_folder (see Kafka Connect Plugins). Connectors listed under plugins are created or updated through the Connect REST API when the worker starts; an entry with enabled: false is removed. The shipped values.yaml carries complete, commented examples for MirrorMaker 2, a Camel S3 sink, a ClickHouse sink and a Snowflake sink.

kafka_connectors:
  - name: cluster
    image: apache/kafka:3.9.1 # Apache images only
    # tags: # extra tags for the worker workload only
    #   cpln/largeDisk: 'true'
    multiZone: true
    cpu: 400m
    memory: 1500Mi
    minCpu: 100m
    minMemory: 375Mi
    plugins_folder: /opt/kafka/plugins
    timeoutSeconds: 15
    replicas: 1
    verbose: false # extra downloader log lines; URLs are always redacted
    extraVolumes: []
    volumes: # the plugins volume set; all keys optional
      initialCapacity: 10 # In GB
      performanceClass: general-purpose-ssd
      fileSystemType: ext4
      snapshots:
        createFinalSnapshot: true
        retentionDuration: 7d
    env:
      - name: KAFKA_HEAP_OPTS
        value: '-Xms900m -Xmx900m' # 50%-75% of memory
    firewall:
      internal_inboundAllowType: "same-gvc"
      external_outboundAllowCIDR: "0.0.0.0/0" # the downloader needs outbound access to fetch artifacts
    listener: client # the kafka.listeners entry the worker connects through
    connector_properties: # written to connect-distributed.properties; the chart fills bootstrap.servers from the listener
      group.id: "connect-cluster"
      security.protocol: "SASL_PLAINTEXT"
      sasl.mechanism: "PLAIN"
      sasl.jaas.config: "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"admin\" password=\"your-admin-password\";"
      offset.storage.topic: "connect-offsets"
      offset.storage.replication.factor: "3"
      config.storage.topic: "connect-configs"
      config.storage.replication.factor: "3"
      status.storage.topic: "connect-status"
      status.storage.replication.factor: "3"
      plugin.path: "/opt/kafka/plugins"
      key.converter: "org.apache.kafka.connect.storage.StringConverter"
      value.converter: "org.apache.kafka.connect.converters.ByteArrayConverter"
    plugins:
      - name: "camel-s3-sink"
        enabled: true
        artifacts:
          - type: tgz # jar / zip / tar / tgz / tar.gz
            url: https://repo.maven.apache.org/maven2/org/apache/camel/kafkaconnector/camel-aws-s3-sink-kafka-connector/4.8.5/camel-aws-s3-sink-kafka-connector-4.8.5-package.tar.gz
        config:
          "connector.class": "org.apache.camel.kafkaconnector.awss3sink.CamelAwss3sinkSinkConnector"
          "tasks.max": "1"
          "topics": "your-topic"

Kafka Connect Plugins

Every Kafka Connect replica runs two containers: kafka-connect and plugins-downloader. The downloader fetches every artifact of each plugin that has enabled: true onto the plugins volume set. Kafka Connect scans its plugins folder only once per start, so it waits until every artifact is in place before it starts. Each artifact is downloaded to a staging area on the volume, verified, and only then moved into its own directory (<plugins_folder>/<plugin>/<key>/), so in a normal start Kafka Connect never sees a half-written file. An artifact that is already installed and unchanged is not downloaded again, so a restart does not repeat the downloads.

kafka_connectors:
  - name: cluster
    plugins_wait_timeout_seconds: 900 # how long Kafka Connect waits for plugin downloads; 0 = wait forever
    plugins_wait_timeout_action: start # start = start without the missing plugins (logged as WARNING) / restart = exit and wait again
    plugins_redownload_token: "" # change to any new value to download every plugin again
    downloader_image: busybox:1.37.0-musl # a custom image must provide the busybox tools listed in values.yaml
    downloader_cpu: 80m # set 250m for artifacts of 50 MB or more
    plugins:
      - name: snowflake-sink
        enabled: true # required for a download; without it the connector is still created
        artifacts:
          - type: jar # jar / zip / tar / tgz / tar.gz
            url: https://repo1.maven.org/maven2/com/snowflake/snowflake-kafka-connector/3.1.1/snowflake-kafka-connector-3.1.1.jar
            sha256: <64 hex characters> # optional: verified before install

enabled: true is required for a download. A plugin without the key is not downloaded, but its connector is still created; use that for a plugin you placed in the plugins folder yourself. With enabled: false the connector is deleted, and the artifacts the chart downloaded for it are removed.
Set sha256 on every artifact. The downloader does not verify TLS certificates, so the checksum is the integrity check. On a mismatch the artifact is not installed and the download is retried. Without sha256, a jar or archive is still checked to be a valid file before it is installed.
Each artifact gets its own directory, so a plugin with several jar artifacts keeps all of them, and they load in one classloader.
Files you placed on the volume yourself are never deleted. The chart removes only artifacts it downloaded, only after a download pass succeeds, and nothing while any artifact has a configuration error (an unsupported type, a missing URL). The downloader lists the files it leaves alone at each start (left untouched (not downloaded by this chart)).
Changing an artifact’s url or sha256 downloads that artifact again; the old copy is removed once the pass succeeds. Size the volume for every installed artifact plus room to download and unpack the largest one.
plugins_wait_timeout_seconds and plugins_wait_timeout_action decide what happens when a download is still missing at the timeout: start starts Kafka Connect without it and logs a WARNING naming it; restart exits so the container restarts and waits again. See Troubleshooting.
plugins_redownload_token forces a fresh download of every plugin: set it to a new value and upgrade the release.
downloader_cpu defaults to 80m. Raise it to 250m when an artifact is 50 MB or larger, so the download finishes well inside the wait timeout.
Credentials in an artifact URL (a user:password@ prefix on the host name) are sent only to that URL’s host; a redirect to another host, such as a presigned storage URL, is followed without them. They are redacted in the logs, but they are stored in the RELEASE_NAME-connect-<name>-download secret and in the Helm release. Percent-encode /, ?, # and @ inside the user name or password.
The Kafka Connect workload reports ready while it waits for plugins. Read the kafka-connect log for Waiting for plugin downloads to see that it is still waiting.

Kafka Client

kafka_client: # Kafka CLI tools for testing from inside the GVC; set the whole block to null to omit the workload
  name: client # workload is RELEASE_NAME-client
  image: apache/kafka:3.9.1
  cpu: 500m
  memory: 1000Mi
  firewall:
    # internal_inboundAllowType: "same-gvc" # same-org / same-gvc
    # external_inboundAllowCIDR: 0.0.0.0/0
    external_outboundAllowCIDR: "0.0.0.0/0"

Connecting

What	Address	Notes
Kafka bootstrap server (inside the GVC)	`RELEASE_NAME-cluster.GVC_NAME.cpln.local:9092`	`security.protocol=SASL_PLAINTEXT`, `sasl.mechanism=PLAIN`, credentials from `kafka.listeners.client.sasl`
Kafka public listener	`<replica>-<location>.<publicAddress>:<containerPort>` per broker (direct replica routing) or `<publicAddress>:3000`, `:3001`, … (multi-port)	`security.protocol=SASL_SSL`; users from the public listener’s `sasl.users`
Kafbat UI	`status.canonicalEndpoint` of `RELEASE_NAME-kafbat-ui`, or `kafbat_ui.domain`	Login from `spring.security.user` in your configuration secret
REST Proxy	`status.canonicalEndpoint` of `RELEASE_NAME-rest-proxy`, `kafka_rest_proxy.domain`, or `RELEASE_NAME-rest-proxy.GVC_NAME.cpln.local:8082` inside the GVC	HTTP BASIC auth with a user from `kafka_rest_proxy.password_properties`
Kafka Connect REST API	`RELEASE_NAME-connect-<name>.GVC_NAME.cpln.local:8083` inside the GVC	Standard Connect REST API
Stored credentials	`cpln secret reveal RELEASE_NAME-secrets -o yaml`	KRaft cluster ID and the admin, controller and inter-broker passwords

Read a canonical endpoint from the workload:

cpln workload get RELEASE_NAME-kafbat-ui --gvc GVC_NAME -o yaml   # status.canonicalEndpoint

Verify the cluster from the bundled client workload. Open a shell on it, write a client properties file with the admin credentials, then list topics, produce and consume:

cpln workload connect RELEASE_NAME-client --gvc GVC_NAME

cat > /tmp/client.properties <<'EOF'
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="ADMIN_PASSWORD";
EOF

/opt/kafka/bin/kafka-topics.sh --bootstrap-server RELEASE_NAME-cluster.GVC_NAME.cpln.local:9092 \
  --command-config /tmp/client.properties --list

/opt/kafka/bin/kafka-console-producer.sh --bootstrap-server RELEASE_NAME-cluster.GVC_NAME.cpln.local:9092 \
  --producer.config /tmp/client.properties --topic test-topic

/opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server RELEASE_NAME-cluster.GVC_NAME.cpln.local:9092 \
  --consumer.config /tmp/client.properties --topic test-topic --from-beginning

With a PLAINTEXT client listener, drop the --command-config, --producer.config and --consumer.config flags. Topics are created on first use because auto.create.topics.enable is true. To check the REST Proxy from a workload in the same GVC, list the clusters it sees (the v3 API) with one of the BASIC-auth users you configured:

curl -u user:USER_PASSWORD http://RELEASE_NAME-rest-proxy.GVC_NAME.cpln.local:8082/v3/clusters

Operations

Backing Up

Kafka’s first line of defence is replication: with the default three brokers, every topic is replicated three times and a write is acknowledged only once two replicas have it. On top of that, each broker volume set takes a snapshot daily at 00:00 UTC, keeps it for seven days and takes a final snapshot when the volume set is deleted (kafka.volumes.logs.snapshots). Take an extra snapshot before risky changes, once per log-directory volume set:

cpln volumeset snapshot create RELEASE_NAME-logs-0 --gvc GVC_NAME --snapshot-name pre-upgrade
cpln volumeset snapshot create RELEASE_NAME-logs-1 --gvc GVC_NAME --snapshot-name pre-upgrade
cpln volumeset snapshot get RELEASE_NAME-logs-0 --gvc GVC_NAME

Restoring a Backup

Restoring a volume snapshot onto a running broker has not been verified for this template; a broker whose log directories are replaced underneath it, or whose KRaft metadata no longer matches its peers, is not something this page can promise will rejoin cleanly. Treat snapshots as a last resort and rely on replication for broker loss. What the chart does support is mounting volume sets that were restored or renamed outside the chart: list them, in kafka.logDirs order, and the chart stops creating its own and mounts yours instead, leaving their data and settings untouched. Use "" for any entry the chart should keep managing.

kafka:
  logDirs: /opt/kafka/logs-0,/opt/kafka/logs-1
  volumes:
    logs:
      externalVolumeSets:
        - kafka-logs-0-fresh
        - kafka-logs-1-fresh

The list must have exactly one entry per kafka.logDirs path, or rendering fails with an error naming both counts.

Upgrading from 2.x

Since 3.0.0 the chart only supports Apache Kafka images. A values file that still sets a Bitnami image fails to render with Error: This chart does not support Bitnami images, please use Apache Kafka images instead. Set kafka.image to an apache/kafka tag. Moving the data of a 2.x Bitnami-based release onto 3.x or later has not been verified; plan for a new release and re-mirror topics (the kafka_connectors MirrorMaker 2 example in values.yaml is one way).

Upgrading from 3.x

4.0.0 added the kafka-orchestrator readiness sidecar (set kafka_orchestrator: null to opt out), raised the graceful shutdown window to kafka.terminationGracePeriodSeconds: 600, and started deriving default.replication.factor and min.insync.replicas from kafka.replicas. A default.replication.factor you carried over in kafka.extra_configurations still wins; remove it unless you need a non-default value.
4.0.2 commented out the sample REST Proxy user. If you relied on the default user BASIC-auth account, set kafka_rest_proxy.password_properties explicitly.
4.1.0 added kafka.rackAwareness, kafka.volumes.logs.externalVolumeSets and per-connector tags; all off by default, nothing to change.
4.2.0 stops the brokers writing a duplicate server.log to the container filesystem on apache/kafka:3.x images by setting KAFKA_LOG4J_ROOT_LOGLEVEL=INFO; the broker log is in cpln logs. The upgrade restarts the brokers.

Upgrading from 4.2.x

4.3.0 changes only how Kafka Connect gets its plugins: 4.2.x slept a fixed 60 seconds before starting, so a slow download could leave a connector missing or FAILED; 4.3.0 waits for the downloads to finish (see Kafka Connect Plugins). Nothing in your values has to change.

The upgrade restarts every Kafka Connect replica, one at a time, and the brokers.
The first start after the upgrade downloads every plugin again, once per replica, into the new layout, and Kafka Connect waits for it. Expect each replica’s connectors to be down for about one full download. Later restarts reuse the installed plugins.
The chart’s old <plugin>/<plugin>.jar files, for plugins that still have a jar artifact, are moved to .cpln-downloads/retired/ in the plugins folder. A file you placed at exactly that path is moved too, and can be moved back.
Directories that earlier versions extracted from zip, tar or tgz artifacts stay in place, and Kafka Connect still loads them beside the new copy. Once your connectors are RUNNING, remove them on each replica (RELEASE_NAME-connect-<name>-0, -1, …), then redeploy. The candidates are the entries the downloader lists at start as left untouched (not downloaded by this chart), minus any you placed yourself; directories named after a plugin’s name, .cpln-downloads and lost+found are never candidates. The commands below are for a connector entry named cluster; in a GVC with more than one location, add --location LOCATION to each exec:
cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-connect-cluster"} |= "left untouched"' --limit 50 --since 24h cpln workload exec RELEASE_NAME-connect-cluster --gvc GVC_NAME --replica RELEASE_NAME-connect-cluster-0 --container plugins-downloader -- ls -A /opt/kafka/plugins cpln workload exec RELEASE_NAME-connect-cluster --gvc GVC_NAME --replica RELEASE_NAME-connect-cluster-0 --container plugins-downloader -- rm -rf /opt/kafka/plugins/OLD_DIRECTORY cpln workload force-redeployment RELEASE_NAME-connect-cluster --gvc GVC_NAME
4.2.x printed artifact URLs, including any credentials in them, in the downloader log; 4.3.0 redacts them. Consider rotating any credential that was part of an artifact URL.
Rolling back to 4.2.x works and the connectors keep running, but 4.2.x writes its <plugin>/<plugin>.jar files again beside the 4.3.0 directories (both load in one classloader) and its 60-second startup race returns. Upgrading to 4.3.0 again retires those files.

Scaling and Availability

kafka.replicas accepts 1, 3, 4 or 5. Rendering fails with Invalid value for kafka.replicas. It must be less than or equal to 5 and not equal to 2. otherwise. Every broker is also a KRaft controller, so never shrink a running cluster below the number of controllers it was formed with — the quorum is lost and the cluster stops.
A single-broker cluster replicates nothing; use it for development only.
kafka.multiZone: true spreads the brokers across availability zones. On AWS, add kafka.rackAwareness.enabled: true and set client.rack on consumers to their zone ID (workloads on AWS locations receive it in the AWS_ZONE_ID environment variable) so they fetch from a same-zone replica; producers and replication are unaffected.
Broker volumes grow automatically up to kafka.volumes.logs.autoscaling.maxCapacity when free space drops below minFreePercentage.
Kafbat UI, the REST Proxy and each Kafka Connect worker scale with their own replicas value.

Troubleshooting

Broker output is in cpln logs; filter server-side:

cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-cluster"} |= "ERROR"' --limit 50 --since 10m

Kafbat UI never becomes ready and cpln logs shows nothing

Cause: the configuration secret named in kafbat_ui.configuration_secret did not exist when the workload was created. The container never starts, so there is no log output.Fix: read status.versions[].message from cpln workload get-deployments RELEASE_NAME-kafbat-ui --gvc GVC_NAME -o yaml to see which secret is missing, create it as shown in Prerequisites, then wait a few minutes or run cpln workload force-redeployment RELEASE_NAME-kafbat-ui --gvc GVC_NAME.

Install fails with Invalid value for kafka.replicas

Cause: kafka.replicas is 2 or more than 5. Two controllers cannot form a quorum, and the chart caps the cluster at five.Fix: set kafka.replicas to 1, 3, 4 or 5.

Install fails with This chart does not support Bitnami images

Cause: kafka.image, kafka_client.image or a connector image references a Bitnami image, which the chart stopped supporting in 3.0.0.Fix: use an apache/kafka tag such as apache/kafka:3.9.1.

Install fails with containerPort cannot be 9091, 9093, or 9094 for listener

Cause: a listener’s containerPort (or directReplicaRouting.containerPort) uses one of the ports reserved for the controller and inter-broker listeners.Fix: pick another port, for example 9092 for the client listener and 9095 for a public one.

Install fails with kafka.volumes.logs.externalVolumeSets has N entries but kafka.logDirs defines M log dirs

Cause: externalVolumeSets is set but its length does not match the number of comma-separated paths in kafka.logDirs. The chart refuses a partial list so it never mounts an empty new volume set beside imported ones.Fix: provide exactly one entry per log directory, in the same order, using "" for any directory the chart should keep managing.

REST Proxy requests fail and its log mentions kafka-dev-cluster

Cause: kafka_rest_proxy.properties.bootstrap.servers still has the shipped default SASL_PLAINTEXT://kafka-dev-cluster:9092, which only resolves for a release named kafka-dev, or the KafkaClient password in kafka_rest_proxy.jaas_conf does not match kafka.listeners.client.sasl.admin.password.Fix: set bootstrap.servers to SASL_PLAINTEXT://RELEASE_NAME-cluster.GVC_NAME.cpln.local:9092, align the KafkaClient password, add at least one user under kafka_rest_proxy.password_properties, and upgrade the release.

A changed Kafbat UI configuration is not picked up

Cause: the configuration secret is read when the container starts; updating the secret does not restart the workload.Fix: run cpln workload force-redeployment RELEASE_NAME-kafbat-ui --gvc GVC_NAME after changing the secret.

Kafka Connect started before its plugins were ready

Cause: an artifact was still missing when plugins_wait_timeout_seconds ran out: a large or slow download, a dead URL or a sha256 mismatch. With the default plugins_wait_timeout_action: start, the kafka-connect log shows WARNING: connector plugins not ready after 900s; starting Kafka Connect WITHOUT: followed by the missing artifacts, and connectors that need them fail. With restart it shows FATAL: connector plugins not ready after 900s; exiting so the container restarts and waits again. and the container restarts. Read the log (for a connector entry named cluster):

cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-connect-cluster"} |= "plugins not ready"' --limit 50 --since 30m
cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-connect-cluster"} |= "ERROR"' --limit 50 --since 30m

Fix: fix any URL or checksum named in an ERROR download failed or ERROR sha256 mismatch line. For large artifacts, raise downloader_cpu to 250m or raise plugins_wait_timeout_seconds; use plugins_wait_timeout_action: restart if Kafka Connect must never run without a plugin. Once the download succeeds, the downloader logs

all artifacts are now installed; Kafka Connect started without some of them, so restart the worker (cpln workload force-redeployment) to load them.

Then run cpln workload force-redeployment RELEASE_NAME-connect-cluster --gvc GVC_NAME. Restarting only the kafka-connect container does not wait again and does not clean up; a redeployment does both.

A connector is FAILED with Failed to find any class

Cause: its plugin arrived after Kafka Connect scanned the plugins folder. Restarting the connector through the REST API does not rescan the folder.Fix: run cpln workload force-redeployment RELEASE_NAME-connect-cluster --gvc GVC_NAME.

Force a plugin re-download

Cause: installed artifacts are reused on every restart, so a plugin whose file changed on the server keeps the copy already on the volume.Fix: set plugins_redownload_token on the connector entry to a new value (any string) and upgrade the release. Every plugin of that entry is downloaded again and the old copies are removed once the download pass succeeds.

Important Notes

Replace every your-… placeholder (kafka.secrets.*, kafka.listeners.client.sasl.*) before installing; they are accepted as literal passwords. They are plain template values and therefore land in the Helm release.
kafka.secrets.kraft_cluster_id is fixed once the brokers have started; changing it later means a new cluster.
kafka.replicas must be 1, 3, 4 or 5. Never shrink a running cluster below its controller count.
Kafbat UI and the REST Proxy are open to the internet by default (external_inboundAllowCIDR: 0.0.0.0/0); set a strong console login and BASIC-auth users, or narrow the CIDR.
The REST Proxy defaults only work for a release named kafka-dev: set kafka_rest_proxy.properties.bootstrap.servers, kafka_rest_proxy.jaas_conf and kafka_rest_proxy.password_properties for your release, or disable it.
ghcr.io/kafbat/kafka-ui, confluentinc/cp-kafka-rest:latest and the JMX exporter image are not pinned, so a redeploy may pick up a newer build; pin a tag in your values if that matters.
Uninstalling deletes the volume sets and every topic’s data; a final snapshot is kept for retentionDuration because createFinalSnapshot is true.
Only set KAFKA_LOG4J_ROOT_LOGLEVEL in kafka.env on apache/kafka:3.x images; on a 4.x image it removes every output from the root logger, including cpln logs.
kafka_ui (Provectus Kafka UI) is deprecated and off by default; use kafbat_ui.
Changing broker values restarts the brokers; schedule upgrades accordingly.
Set sha256 on every Kafka Connect plugin artifact: the plugin downloader does not verify TLS certificates.

External References

Apache Kafka Documentation

Official Apache Kafka documentation

KRaft Mode

Kafka KRaft consensus mode without ZooKeeper

Kafbat UI Configuration

Configuration file reference for the Kafbat UI secret

Confluent REST Proxy API

REST Proxy v2 and v3 API reference

Kafka Template

View the source files, default values, and chart definition

​Overview

​What Gets Created

​Prerequisites

​Installation

UI

CLI

Terraform

Pulumi

​Configuration

​Kafka Cluster

​Storage

​Firewall

​Listeners

​Client Listener

​Public Listener

​ACL and Cluster Secrets

​Extra Configurations

​Health Sidecar and Exporters

​Kafbat UI

​Kafka REST Proxy

​Kafka Connect

​Kafka Connect Plugins

​Kafka Client

​Connecting

​Operations

​Backing Up

​Restoring a Backup

​Upgrading from 2.x

​Upgrading from 3.x

​Upgrading from 4.2.x

​Scaling and Availability

​Troubleshooting

​Important Notes

​External References

Apache Kafka Documentation

KRaft Mode

Kafbat UI Configuration

Confluent REST Proxy API

Kafka Template

Overview

What Gets Created

Prerequisites

Installation

Configuration

Kafka Cluster

Storage

Firewall

Listeners

Client Listener

Public Listener

ACL and Cluster Secrets

Extra Configurations

Health Sidecar and Exporters

Kafbat UI

Kafka REST Proxy

Kafka Connect

Kafka Connect Plugins

Kafka Client

Connecting

Operations

Backing Up

Restoring a Backup

Upgrading from 2.x

Upgrading from 3.x

Upgrading from 4.2.x

Scaling and Availability

Troubleshooting

Important Notes

External References