Elasticsearch

Overview

Elasticsearch is a distributed search and analytics engine built on Apache Lucene. This template deploys an Elasticsearch 8.17.0 cluster whose nodes discover each other and elect a master automatically, a Kibana (on by default) workload for querying data and managing indices, and optional incremental snapshot backups to AWS S3 or GCS driven by Elasticsearch’s built-in Snapshot Lifecycle Management (SLM). The cluster runs with X-Pack security disabled: there is no username, password or TLS anywhere in the template, and the only access control is the internal_access setting on each workload.

This template deploys into an existing GVC that you already have. It does not create or manage a GVC. Every resource lands in the GVC you name at install time, so cpln workload exec, cpln logs and cpln helm uninstall all run against that GVC, and uninstalling the release never deletes it.

What Gets Created

Resource	Name	Purpose	Optional
Workload	`RELEASE_NAME-elasticsearch`	Elasticsearch nodes (stateful, `replicas` replicas, HTTP on 9200, transport on 9300)	—
Workload	`RELEASE_NAME-kibana`	Kibana web UI (standard, 1 replica, port 5601)	`kibana.enabled: true` (default)
Workload	`RELEASE_NAME-backup-setup`	Setup job that registers the snapshot repository and the SLM policy	`backup.enabled: true`
Volume Set	`RELEASE_NAME-elasticsearch-vs`	One ext4 data volume per node, with 7-day volume snapshots and a final snapshot on deletion	—
Secret	`RELEASE_NAME-elasticsearch-startup`	Node startup script that writes `elasticsearch.yml`	—
Identity	`RELEASE_NAME-elasticsearch-identity`	Bound to every workload; carries the cloud account binding when backups are enabled	—
Policy	`RELEASE_NAME-elasticsearch-policy`	Grants the identity `reveal` on the startup secret	—

Prerequisites

None for a default install. Backups are optional and need a bucket and a Control Plane cloud account (plus an IAM policy on AWS) before they can be enabled. The per-provider steps are under Backing Up.

Installation

A default install needs no --set values:

cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/elasticsearch \
  --version 1.0.1 --gvc GVC_NAME

To install with a different method, follow the instructions for your preferred tool:

UI

Browse, install, and manage templates visually

CLI

Manage templates from your terminal

Terraform

Declare templates in your Terraform configurations

Pulumi

Declare templates in your Pulumi programs

Configuration

Cluster

image: docker.elastic.co/elasticsearch/elasticsearch:8.17.0

replicas: 3 # Must be odd for master quorum

clusterName: my-elasticsearch-cluster

# JVM heap size for each node - set to ~50% of maxMemory, hard cap at 30g
jvmHeap: 3g

Elasticsearch requires an odd number of master-eligible nodes to hold a quorum; the template refuses to render an even replicas value. Three nodes survive the loss of one. jvmHeap is not derived from resources — if you raise resources.maxMemory, decide separately whether to raise the heap, and keep it at roughly half the memory limit so the rest is available to the filesystem cache.

Resources

resources:
  minCpu: 1
  minMemory: 2Gi
  maxCpu: 2
  maxMemory: 6Gi

Volume Set

volumeset:
  capacity: 10 # initial capacity in GiB (minimum is 10)
  autoscaling:
    enabled: false # Set to true to enable autoscaling
    maxCapacity: 100 # Maximum capacity in GiB when autoscaling is enabled
    minFreePercentage: 10 # Minimum free percentage to trigger scaling when autoscaling is enabled
    scalingFactor: 1.2 # Scaling factor to determine how much to scale up when autoscaling is triggered

Each node gets its own volume of volumeset.capacity GiB. With autoscaling enabled, a volume grows by scalingFactor whenever its free space drops below minFreePercentage, up to maxCapacity.

Multi-Zone

multiZone:
  enabled: false # Set to true to schedule replicas across availability zones for higher durability

When enabled, replicas are spread across availability zones within each location, so losing one zone leaves the remaining nodes holding quorum. Confirm the location you deploy to supports multiple availability zones before enabling it.

Internal Access

internal_access:
  type: same-gvc # options: same-gvc, same-org, workload-list
  workloads: # Note: can only be used if type is same-gvc or workload-list
    #- //gvc/GVC_NAME/workload/WORKLOAD_NAME
    #- //gvc/GVC_NAME/workload/WORKLOAD_NAME

Type	Who can reach Elasticsearch
`same-gvc`	Every workload in the same GVC (default)
`same-org`	Every workload in the org
`workload-list`	Only the workloads listed under `workloads`

Because the cluster has no authentication, this is the only thing standing between a workload and full admin access to every index. With workload-list, remember to list the template’s own consumers: //gvc/GVC_NAME/workload/RELEASE_NAME-kibana when Kibana is enabled, and //gvc/GVC_NAME/workload/RELEASE_NAME-backup-setup when backups are enabled — both reach Elasticsearch over this port.

Kibana

kibana:
  enabled: true
  image: docker.elastic.co/kibana/kibana:8.17.0
  resources:
    cpu: 500m
    memory: 2Gi
  internal_access:
    type: same-gvc # options: same-gvc, same-org, workload-list
    workloads: # Note: can only be used if type is same-gvc or workload-list
      #- //gvc/GVC_NAME/workload/WORKLOAD_NAME

Kibana connects to RELEASE_NAME-elasticsearch.GVC_NAME.cpln.local:9200 and has its own kibana.internal_access block with the same three options as Elasticsearch.

Backup

backup:
  enabled: false
  remove_setup_workload: false # Set to true after backup setup is no longer needed to reduce resource usage

  # Cloud provider: aws or gcp
  provider: aws

  # Snapshot schedule in Quartz cron format (6 fields: seconds minutes hours day-of-month month day-of-week)
  # e.g. "0 0 2 * * ?" = daily at 2am UTC. Note: ES SLM requires Quartz format, not standard 5-field cron.
  schedule: "0 0 2 * * ?"

  # Snapshot retention
  retention:
    maxAge: 30d   # Delete snapshots older than this
    maxCount: 30  # Keep at most this many snapshots

  # AWS S3 Configuration
  aws:
    bucket: my-s3-bucket               # S3 bucket name (required if provider=aws)
    region: us-east-1                  # S3 bucket region (required if provider=aws)
    prefix: elasticsearch-snapshots    # Path prefix in bucket
    cloudAccountName: my-cloud-account # Control Plane Cloud Account name (required)
    policyName: my-backup-policy       # AWS IAM custom policy name (required)

  # GCP GCS Configuration
  gcp:
    bucket: my-gcs-bucket              # GCS bucket name (required if provider=gcp)
    prefix: elasticsearch-snapshots    # Path prefix in bucket
    cloudAccountName: my-cloud-account # Control Plane Cloud Account name (required)

The bucket, cloud account and (on AWS) IAM policy must exist before you enable backups — see Backing Up for the per-provider steps and what the setup job does.

Connecting

Both workloads are reachable only from inside the GVC by default; external inbound traffic is closed and there is no public endpoint. Use cpln port-forward to reach either from your machine.

What	Address	Notes
Elasticsearch REST API (same GVC)	`http://RELEASE_NAME-elasticsearch.GVC_NAME.cpln.local:9200`	Plain HTTP, no credentials
Elasticsearch transport	`RELEASE_NAME-elasticsearch.GVC_NAME.cpln.local:9300`	Node-to-node traffic only
Kibana (same GVC)	`http://RELEASE_NAME-kibana.GVC_NAME.cpln.local:5601`	When `kibana.enabled: true`
Public endpoint	None	Reach the UI and API with `cpln port-forward`
Credentials	None	X-Pack security is disabled; `internal_access` is the only access control

Always use the fully qualified hostname; the bare workload name is not reliably resolvable. Check cluster health from inside an Elasticsearch container:

cpln workload exec RELEASE_NAME-elasticsearch --gvc GVC_NAME --container elasticsearch -- curl -s 'http://localhost:9200/_cluster/health?pretty'

Open Kibana or the REST API in a browser or local client:

cpln port-forward RELEASE_NAME-kibana 5601:5601 --gvc GVC_NAME
# then open http://localhost:5601

cpln port-forward RELEASE_NAME-elasticsearch 9200:9200 --gvc GVC_NAME
# then curl http://localhost:9200

Operations

Backing Up

Backups are incremental Elasticsearch snapshots written straight to S3 or GCS by the repository-s3 and repository-gcs modules bundled in the official image — no separate backup image runs. Snapshot Lifecycle Management (SLM) takes them on backup.schedule and prunes them by backup.retention. When backup.enabled: true, the template adds the RELEASE_NAME-backup-setup workload. It waits for the cluster to answer /_cluster/health, registers the snapshot repository as backup-repo, verifies the repository can reach the bucket, and creates an SLM policy named automated-snapshots that snapshots every index except Elasticsearch’s internal and history indices with include_global_state: false. The job is idempotent, so it is safe for it to run more than once. Watch it finish:

cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-backup-setup"}' --limit 50 --since 10m

Once the logs show Backup Configuration Complete!, remove the job to free its resources:

backup:
  enabled: true
  remove_setup_workload: true

backup.schedule uses Quartz cron format with 6 fields (seconds, minutes, hours, day-of-month, month, day-of-week), not the 5-field cron most schedulers use. "0 0 2 * * ?" runs daily at 02:00 UTC; the ? is required in day-of-week when day-of-month is set.

Enabling backups on a cluster that is already running can take several minutes to settle. The setup job fails and restarts until every Elasticsearch node can reach the bucket, logging ERROR: Failed to register repository in the meantime. Keep watching its logs rather than reading the first error as a misconfiguration.

AWS S3

Create the bucket

Create an S3 bucket. Set backup.aws.bucket to its name and backup.aws.region to its region.

Create or pick a cloud account

If you do not have one, create a cloud account for your AWS account. Set backup.aws.cloudAccountName to its name.

Create a bucket-scoped IAM policy

Create an IAM policy with the following JSON, replacing YOUR_BUCKET_NAME. Both resource ARNs are required: s3:ListBucket and s3:GetBucketLocation authorize against the bucket itself, the rest against its objects.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:ListBucket",
                "s3:GetObjectVersion",
                "s3:DeleteObjectVersion",
                "s3:GetBucketLocation",
                "s3:AbortMultipartUpload",
                "s3:ListBucketMultipartUploads",
                "s3:ListMultipartUploadParts"
            ],
            "Resource": [
                "arn:aws:s3:::YOUR_BUCKET_NAME",
                "arn:aws:s3:::YOUR_BUCKET_NAME/*"
            ]
        }
    ]
}

Set backup.aws.policyName to the policy’s name. The template attaches it, together with the platform’s cpln-connector policy, to the release identity.

GCS

Create the bucket

Create a GCS bucket. Set backup.gcp.bucket to its name.

Create or pick a cloud account

If you do not have one, create a cloud account for your GCP project. Set backup.gcp.cloudAccountName to its name.

Grant storage access

Add the Storage Admin role to the GCP service account associated with the cloud account. The template additionally binds roles/storage.objectAdmin on the bucket to the release identity.

Manual Snapshots

With a port-forward to Elasticsearch open (cpln port-forward RELEASE_NAME-elasticsearch 9200:9200 --gvc GVC_NAME), you can trigger a snapshot or inspect the repository directly:

# Take a snapshot now using the SLM policy
curl -X PUT 'http://localhost:9200/_slm/policy/automated-snapshots/_execute'

# List every snapshot in the repository
curl 'http://localhost:9200/_snapshot/backup-repo/_all?pretty'

# Show snapshots currently in progress
curl 'http://localhost:9200/_snapshot/backup-repo/_current?pretty'

# Show the SLM policy and its last run
curl 'http://localhost:9200/_slm/policy/automated-snapshots?pretty'

Restoring a Backup

The commands below follow the upstream Elasticsearch snapshot and restore API and use the repository name this template registers (backup-repo). They have not been exercised end to end against this template. Rehearse a restore on a non-production release before you depend on it.

All restores run through the REST API. Open a port-forward first (cpln port-forward RELEASE_NAME-elasticsearch 9200:9200 --gvc GVC_NAME), then list what is available:

curl 'http://localhost:9200/_snapshot/backup-repo/_all?pretty'

Disaster recovery into a fresh cluster. Install a new release with backup.enabled: true and the same bucket, prefix and provider as the lost cluster. The setup job registers the same repository, so the old snapshots become visible. Then restore everything except Elasticsearch’s internal indices:

curl -X POST 'http://localhost:9200/_snapshot/backup-repo/SNAPSHOT_NAME/_restore' \
  -H 'Content-Type: application/json' \
  -d '{
    "indices": ["*", "-.internal.*", "-.slo-observability.*.temp", "-.ds-ilm-history*"],
    "ignore_unavailable": true,
    "include_global_state": false
  }'

Restoring an index that still exists. Elasticsearch refuses to restore over an open index with the same name. Close it first, then restore:

curl -X POST 'http://localhost:9200/MY_INDEX/_close'

curl -X POST 'http://localhost:9200/_snapshot/backup-repo/SNAPSHOT_NAME/_restore' \
  -H 'Content-Type: application/json' \
  -d '{
    "indices": "MY_INDEX",
    "ignore_unavailable": true,
    "include_global_state": false
  }'

Restoring specific indices takes a comma-separated list with wildcards:

curl -X POST 'http://localhost:9200/_snapshot/backup-repo/SNAPSHOT_NAME/_restore' \
  -H 'Content-Type: application/json' \
  -d '{
    "indices": "my-index,my-other-index-2026.05*",
    "ignore_unavailable": true,
    "include_global_state": false
  }'

Monitor the restore until the cluster returns to green:

curl 'http://localhost:9200/_cat/recovery?v&active_only=true'
curl 'http://localhost:9200/_cluster/health?pretty'

Upgrading from 1.0.0

Version 1.0.1 removes the AWS managed policy aws::ReadOnlyAccess from the backup identity. That policy granted read access to every bucket in your AWS account and contained no write actions, so it never carried the backup itself — but it was silently supplying any read action your own bucket-scoped policy happened to omit. The identity now carries cpln-connector and your backup.aws.policyName policy only. Before upgrading an AWS-backed release, compare your IAM policy with the full action list under Backing Up and add anything missing. If it already matches, no action is needed. Releases without backups, and GCS-backed releases, are unaffected.

Scaling and Availability

Scaling up. Raise replicas to the next odd number and run cpln helm upgrade. The new nodes join the cluster automatically and Elasticsearch begins rebalancing shards onto them.

Scaling down is destructive. Elasticsearch does not move shards off a node before the node is removed. Take a snapshot first, then check GET /_cat/shards?v to confirm no primary shard would be stranded on the nodes you are about to remove, and only then lower replicas.

Changing replicas, jvmHeap, resources or image restarts the Elasticsearch nodes. Expect the cluster to report yellow while shards recover after a restart; reads and writes keep working as long as a quorum of nodes is up. With multiZone.enabled: true, replicas are spread across availability zones so the loss of a zone does not cost the quorum.

Troubleshooting

Install or upgrade fails with 'replicas must be a positive odd number'

Symptom: cpln helm install or cpln helm upgrade stops before creating anything with replicas must be a positive odd number, got: 4.Cause: The template validates replicas at render time because Elasticsearch needs an odd number of master-eligible nodes to elect a master.Fix: Set replicas to 1, 3, 5 or another odd number and rerun the command.

Backup setup job logs 'ERROR: Failed to register repository'

Symptom: cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-backup-setup"}' --limit 50 --since 10m shows ERROR: Failed to register repository (HTTP 500) followed by the Elasticsearch response body.Cause: Elasticsearch could not write to the bucket. On AWS the usual reasons are a wrong backup.aws.bucket or backup.aws.region, an IAM policy missing one of the actions listed under Backing Up, or a cloud account not yet linked. On GCP it is usually a missing Storage Admin role. Right after enabling backups on a running cluster the same error appears for a few minutes while the nodes pick up the new configuration.Fix: Read the response body printed under the error — Elasticsearch names the failing permission or bucket. Correct the IAM policy or value, and run cpln helm upgrade if a value changed. The job restarts on its own and succeeds once the bucket is reachable.

Restore is refused because an open index with the same name already exists

Symptom: The _restore call returns Cannot restore index [MY_INDEX] because an open index with same name already exists in the cluster.Cause: Elasticsearch never overwrites an open index during a restore.Fix: Either close or delete the existing index before restoring, or restore under a new name:

curl -X POST 'http://localhost:9200/_snapshot/backup-repo/SNAPSHOT_NAME/_restore' \
  -H 'Content-Type: application/json' \
  -d '{
    "indices": "MY_INDEX",
    "rename_pattern": "(.+)",
    "rename_replacement": "restored-$1",
    "include_global_state": false
  }'

Elasticsearch containers restart with OOMKilled

Symptom: Nodes keep restarting, and the deployment status shows the restart reason OOMKilled:

cpln workload get-deployments RELEASE_NAME-elasticsearch --gvc GVC_NAME -o yaml

Cause: jvmHeap is too close to resources.maxMemory. The JVM heap is only part of what Elasticsearch uses; the rest of the limit must stay free for off-heap memory and the filesystem cache.Fix: Lower jvmHeap to roughly half of resources.maxMemory, or raise resources.maxMemory, and run cpln helm upgrade.

Kibana shows 'Kibana server is not ready yet'

Symptom: Opening Kibana returns the plain-text page Kibana server is not ready yet.Cause: Kibana cannot reach RELEASE_NAME-elasticsearch.GVC_NAME.cpln.local:9200. Either the cluster is still starting — nodes take a minute or two to become ready — or internal_access.type is workload-list and //gvc/GVC_NAME/workload/RELEASE_NAME-kibana is not in internal_access.workloads.Fix: Wait for the Elasticsearch deployment to report ready, then reload. If it stays unready, add the Kibana workload to internal_access.workloads and run cpln helm upgrade.

Important Notes

There is no authentication. X-Pack security is disabled, so anything internal_access admits has full admin access to every index. Keep internal_access as narrow as your deployment allows.
replicas must be odd (1, 3, 5, …) or the chart refuses to render; an even count cannot hold a master quorum.
jvmHeap does not follow resources.maxMemory. Raising the memory limit does not raise the heap; setting the heap above about half the limit gets the node OOM-killed.
Snapshot before scaling down. Elasticsearch does not evacuate shards from nodes that are about to be removed.
backup.schedule is 6-field Quartz cron, not 5-field cron, and needs ? in the day-of-week field when day-of-month is set.
The AWS IAM policy needs both bucket ARNs — arn:aws:s3:::BUCKET and arn:aws:s3:::BUCKET/*.
Enabling backups on a running cluster takes several minutes to settle; the setup job retries until every node can reach the bucket.

External References

Elasticsearch Reference

Official Elasticsearch reference documentation

Snapshot and Restore

Repositories, SLM policies and how snapshots work

Restore a Snapshot

The restore API, renaming on restore, and recovery monitoring

Quorum-based Decision Making

Why master-eligible node counts should be odd

Elasticsearch Template

View the source files, default values, and chart definition

Template Catalog

Templates

Overview

What Gets Created

Prerequisites

Installation

UI

CLI

Terraform

Pulumi

Configuration

Cluster

Resources

Volume Set

Multi-Zone

Internal Access

Kibana

Backup

Connecting

Operations

Backing Up

AWS S3

GCS

Manual Snapshots

Restoring a Backup

Upgrading from 1.0.0

Scaling and Availability

Troubleshooting

Important Notes

External References

Elasticsearch Reference

Snapshot and Restore

Restore a Snapshot

Quorum-based Decision Making

Elasticsearch Template

​Overview

​What Gets Created

​Prerequisites

​Installation

UI

CLI

Terraform

Pulumi

​Configuration

​Cluster

​Resources

​Volume Set

​Multi-Zone

​Internal Access

​Kibana

​Backup

​Connecting

​Operations

​Backing Up

​AWS S3

​GCS

​Manual Snapshots

​Restoring a Backup

​Upgrading from 1.0.0

​Scaling and Availability

​Troubleshooting

​Important Notes

​External References

Elasticsearch Reference

Snapshot and Restore

Restore a Snapshot

Quorum-based Decision Making

Elasticsearch Template

Overview

What Gets Created

Prerequisites

Installation

Configuration

Cluster

Resources

Volume Set

Multi-Zone

Internal Access

Kibana

Backup

Connecting

Operations

Backing Up

AWS S3

GCS

Manual Snapshots

Restoring a Backup

Upgrading from 1.0.0

Scaling and Availability

Troubleshooting

Important Notes

External References