Redpanda - Control Plane

Overview

Redpanda is a Kafka-compatible streaming platform written in C++. It implements the Kafka wire protocol natively, so any Kafka client, SDK, or tool works without modification. This template deploys a stateful Redpanda broker cluster with SASL authentication, Schema Registry, an optional HTTP REST proxy, and an optional web console.

What Gets Created

Stateful Redpanda Workload — A multi-replica broker cluster using the Seastar async runtime. Each broker gets its own persistent volume.
Standard Redpanda Console Workload (optional, enabled by default) — Web UI for browsing topics, inspecting messages, managing consumer groups, and viewing Schema Registry schemas.
Volume Set — One persistent volume per broker replica for data storage.
Identity & Policy — An identity bound to the workloads with reveal access to credential secrets.
Secrets — A dictionary secret holding SASL user credentials injected at startup.

This template does not create a GVC. You must deploy it into an existing GVC.

Prerequisites

This template has no external prerequisites. To install, follow the instructions for your preferred method:

UI

Browse, install, and manage templates visually

CLI

Manage templates from your terminal

Terraform

Declare templates in your Terraform configurations

Pulumi

Declare templates in your Pulumi programs

Configuration

The default values.yaml for this template:

redpanda:
  name: cluster
  image: redpandadata/redpanda:v26.1.9
  replicas: 3
  multiZone: false
  env: []

  cpu: 1500m
  memory: 4Gi
  minCpu: 500m
  minMemory: 2Gi
  smp: 1           # Seastar reactor threads — must match the floor of your cpu limit
  reserveMemory: 1G # memory reserved for the OS; Redpanda uses (memory - reserveMemory)

  volume:
    initialCapacity: 10 # In GiB
    performanceClass: general-purpose-ssd # or high-throughput-ssd (min 200 GiB)
    fileSystemType: xfs # xfs / ext4
    # customEncryption:
    #   enabled: true
    #   region: aws-us-east-2
    #   keyId: arn:aws:kms:us-east-2:1234567890:key/your-key-id

  firewall:
    internal_inboundAllowType: "same-gvc" # Options: same-org / same-gvc
    # external_inboundAllowCIDR: 0.0.0.0/0
    # inboundAllowWorkload:
    #   - //gvc/my-gvc/workload/my-app

  listeners:
    kafka:
      internal:
        port: 9092
      # external:
      #   directReplicaRouting:
      #     containerPort: 9094
      #     publicAddress: redpanda.example.com

    adminApi:
      port: 9644

    schemaRegistry:
      port: 8081

    pandaproxy:
      enabled: false
      port: 8082

  auth:
    saslMechanism: SCRAM-SHA-256 # SCRAM-SHA-256 / SCRAM-SHA-512
    users:
      - username: admin
        password: "your-admin-password"
      # - username: your-app-user
      #   password: "your-app-password"
    superusers: []

  acl:
    allowEveryoneIfNoAclFound: false

  secrets:
    cluster_id: ""  # leave empty to auto-generate; set explicitly to preserve identity across reinstalls

  extra_configurations: {}
    # auto_create_topics_enabled: false
    # log_retention_ms: 604800000
    # log_segment_size: 134217728

redpanda_console:
  enabled: true
  name: console
  image: redpandadata/console:v3.7.4
  cpu: 200m
  memory: 256Mi
  minCpu: 50m
  minMemory: 64Mi
  replicas: 1
  # domain: console.your-domain.com
  firewall:
    external_inboundAllowCIDR: "0.0.0.0/0"

Cluster Size and Resources

redpanda.replicas — Number of broker replicas. A minimum of 3 is recommended for production to ensure Raft quorum.
redpanda.cpu / redpanda.memory — Maximum CPU and memory per broker.
redpanda.minCpu / redpanda.minMemory — Minimum guaranteed CPU and memory per broker.
redpanda.smp — Number of Seastar reactor threads. Must match the floor of cpu (e.g., cpu: 1500m → smp: 1, cpu: 3 → smp: 3). Without this, Seastar uses all node CPUs and incorrectly divides memory across them.
redpanda.reserveMemory — Memory set aside for the OS. Redpanda uses (memory - reserveMemory) for its own heap. Default 1G works for most configurations.
redpanda.multiZone — Spread brokers across availability zones within the location.

Storage

Each broker replica gets its own persistent volume. For production workloads with high throughput, use high-throughput-ssd (minimum 200 GiB).

redpanda.volume.initialCapacity — Initial volume size in GiB.
redpanda.volume.performanceClass — general-purpose-ssd or high-throughput-ssd.
redpanda.volume.fileSystemType — xfs (default, recommended for Redpanda) or ext4.

Volume encryption via AWS KMS is supported:

redpanda:
  volume:
    customEncryption:
      enabled: true
      region: aws-us-east-2
      keyId: arn:aws:kms:us-east-2:1234567890:key/your-key-id

After deploying with custom encryption enabled, navigate to each created volume in the Control Plane console, click spec, and follow the AWS Custom Encryption Instructions to complete the setup.

Authentication

SASL is always enabled. All users are defined under redpanda.auth.users. The first user in the list is automatically granted superuser privileges. Additional superusers can be added under redpanda.auth.superusers.

redpanda.auth.saslMechanism — SCRAM-SHA-256 (default) or SCRAM-SHA-512.
redpanda.auth.users — List of username / password pairs created at startup.
redpanda.auth.superusers — Additional usernames to grant superuser privileges.

ACLs

redpanda.acl.allowEveryoneIfNoAclFound — When false (default), clients without an explicit ACL are denied. Set to true to allow unauthenticated access when no ACL exists for a resource.

Listeners

Listener	Port	Description
Kafka	`9092`	Kafka-compatible wire protocol (internal)
Admin API	`9644`	Redpanda Admin API for cluster management
Schema Registry	`8081`	Confluent-compatible Schema Registry
PandaProxy	`8082`	HTTP REST proxy (disabled by default)

Enable PandaProxy to produce and consume messages over HTTP without a Kafka client:

redpanda:
  listeners:
    pandaproxy:
      enabled: true
      port: 8082

Extra Broker Configuration

Pass any Redpanda broker property directly via extra_configurations. These are injected into redpanda.yaml at startup:

redpanda:
  extra_configurations:
    auto_create_topics_enabled: false
    log_retention_ms: 604800000    # 7 days
    log_segment_size: 134217728    # 128 MiB
    log_retention_bytes: -1        # unlimited

Firewall

redpanda.firewall.internal_inboundAllowType — Controls which workloads can reach the brokers:
- same-gvc — All workloads in the same GVC (default).
- same-org — All workloads in the org.
redpanda.firewall.inboundAllowWorkload — Allow specific workloads by path.

Connecting

Redpanda is accessible internally from any workload in the same GVC:

Listener	Hostname	Port
Kafka	`{clusterName}.{gvc}.cpln.local`	`9092`
Admin API	`{clusterName}.{gvc}.cpln.local`	`9644`
Schema Registry	`{clusterName}.{gvc}.cpln.local`	`8081`
PandaProxy (if enabled)	`{clusterName}.{gvc}.cpln.local`	`8082`

To connect to a specific broker replica directly:

{clusterName}-0.{clusterName}.{gvc}.cpln.local:9092
{clusterName}-1.{clusterName}.{gvc}.cpln.local:9092

Connect using rpk:

rpk topic list \
  -X brokers={clusterName}.{gvc}.cpln.local:9092 \
  -X sasl.mechanism=SCRAM-SHA-256 \
  -X user=admin \
  -X pass=your-admin-password

For Kafka clients, use the following connection properties:

bootstrap.servers={clusterName}.{gvc}.cpln.local:9092
security.protocol=SASL_PLAINTEXT
sasl.mechanism=SCRAM-SHA-256
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
  username="admin" \
  password="your-admin-password";

Redpanda Console

The Redpanda Console is enabled by default and accessible via the Control Plane external endpoint for the {release-name}-console workload. It provides a web UI for browsing topics, inspecting messages, managing consumer groups, and viewing Schema Registry schemas. To expose the console on a custom domain, set redpanda_console.domain:

redpanda_console:
  domain: console.your-domain.com

This creates a Control Plane domain resource that routes HTTPS traffic to the console workload. The same DNS prerequisites apply as for any Control Plane domain (ownership TXT record and CNAME to the GVC alias). To restrict console access to specific IPs, update redpanda_console.firewall.external_inboundAllowCIDR:

redpanda_console:
  firewall:
    external_inboundAllowCIDR: "203.0.113.0/24"

External Access

Redpanda brokers can be exposed over the internet via TLS using a public domain. Each broker advertises its own per-replica subdomain and Control Plane routes clients to the correct broker using SNI.

Prerequisites

A domain you control with DNS managed by your registrar (e.g. Cloudflare).
Dedicated Load Balancer enabled on your GVC — required for external TCP routing. Enable under GVC settings in the Control Plane console. See Configure Domain documentation.
DNS records added before deploying. Disable proxying (e.g. Cloudflare’s orange cloud) — TCP traffic must pass through directly:

Type	Name	Value
TXT	`_cpln.your-domain.com`	your Control Plane org name or org ID
CNAME	`@`	`{gvcAlias}.cpln.app`
CNAME	`_acme-challenge`	`_acme-challenge.cpln.app`
CNAME	`{clusterName}-0-{location}`	`{gvcAlias}.cpln.app`
CNAME	`{clusterName}-1-{location}`	`{gvcAlias}.cpln.app`
CNAME	`{clusterName}-N-{location}`	`{gvcAlias}.cpln.app`

Add one CNAME per broker replica. The _acme-challenge record is required for Control Plane to issue the TLS certificate via DNS-01. Your GVC alias is visible under GVC settings in the Control Plane console.

Configuration

redpanda:
  listeners:
    kafka:
      external:
        directReplicaRouting:
          containerPort: 9094
          publicAddress: your-domain.com

Connecting Externally

Each broker advertises its own subdomain in the format {clusterName}-{ordinal}-{location}.{domain}. Use all broker addresses as the bootstrap list:

rpk topic list \
  -X brokers=cluster-0-aws-us-east-1.your-domain.com:9094,cluster-1-aws-us-east-1.your-domain.com:9094,cluster-2-aws-us-east-1.your-domain.com:9094 \
  -X tls.enabled=true \
  -X sasl.mechanism=SCRAM-SHA-256 \
  -X user=admin \
  -X pass=your-admin-password

For Kafka clients:

security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-256
bootstrap.servers=cluster-0-aws-us-east-1.your-domain.com:9094,cluster-1-aws-us-east-1.your-domain.com:9094,cluster-2-aws-us-east-1.your-domain.com:9094
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
  username="admin" \
  password="your-admin-password";

External References

Redpanda Documentation

Official Redpanda documentation

Redpanda Console Documentation

Redpanda Console UI guide

rpk CLI Reference

rpk command reference for managing Redpanda clusters

Schema Registry API

Confluent-compatible Schema Registry and HTTP Proxy API reference

​Overview

​What Gets Created

​Prerequisites

UI

CLI

Terraform

Pulumi

​Configuration

​Cluster Size and Resources

​Storage

​Authentication

​ACLs

​Listeners

​Extra Broker Configuration

​Firewall

​Connecting

​Redpanda Console

​External Access

​Prerequisites

​Configuration

​Connecting Externally

​External References

Redpanda Documentation

Redpanda Console Documentation

rpk CLI Reference

Schema Registry API

Overview

What Gets Created

Prerequisites

Configuration

Cluster Size and Resources

Storage

Authentication

ACLs

Listeners

Extra Broker Configuration

Firewall

Connecting

Redpanda Console

External Access

Prerequisites

Configuration

Connecting Externally

External References