> ## Documentation Index
> Fetch the complete documentation index at: https://docs.controlplane.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Elasticsearch

## Overview

Elasticsearch is a distributed search and analytics engine built on Apache Lucene. This template deploys a production-ready Elasticsearch 8.17.0 cluster with automatic master election, optional Kibana for visualization and management, and automated snapshot backups to AWS S3 or GCS via Elasticsearch's built-in Snapshot Lifecycle Management (SLM).

<Note>
  This template does not create a GVC. You must deploy it into an existing GVC.
</Note>

### What Gets Created

* **Stateful Elasticsearch Workload** — A multi-node Elasticsearch cluster. Each replica gets its own persistent volume so index data and shards survive restarts.
* **Volume Set** — One persistent volume per replica for Elasticsearch data.
* **Standard Kibana Workload** *(optional, enabled by default)* — The Elasticsearch web UI for querying data, managing indices, and monitoring cluster health.
* **Standard Backup Setup Workload** *(optional)* — A one-time job that waits for the cluster to be healthy, registers the snapshot repository with Elasticsearch, and creates the SLM policy. Runs once and can be removed after initial setup.
* **Identity & Policy** — An identity bound to the workloads with `reveal` access to the configuration secrets, and cloud storage access when backup is enabled.
* **Secrets** — Configuration secrets for Elasticsearch and Kibana credentials and cluster settings.

## Prerequisites

This template has no external prerequisites unless backup is enabled. To install, follow the instructions for your preferred method:

<CardGroup cols={2}>
  <Card title="UI" href="/template-catalog/install-manage/ui" icon="laptop">
    Browse, install, and manage templates visually
  </Card>

  <Card title="CLI" href="/template-catalog/install-manage/cli" icon="terminal">
    Manage templates from your terminal
  </Card>

  <Card title="Terraform" href="/template-catalog/install-manage/terraform" icon={<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 128 128"><g fill-rule="evenodd"><path d="M77.941 44.5v36.836L46.324 62.918V26.082zm0 0" fill="#5c4ee5"/><path d="M81.41 81.336l31.633-18.418V26.082L81.41 44.5zm0 0" fill="#4040b2"/><path d="M11.242 42.36L42.86 60.776V23.941L11.242 5.523zm0 0M77.941 85.375L46.324 66.957v36.82l31.617 18.418zm0 0" fill="#5c4ee5"/></g></svg>}>
    Declare templates in your Terraform configurations
  </Card>

  <Card
    title="Pulumi"
    href="/template-catalog/install-manage/pulumi"
    icon={<svg xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 24 24" id="Pulumi-Icon--Streamline-Svg-Logos" height="24" width="24">
    <desc>
        Pulumi Icon Streamline Icon: https://streamlinehq.com
    </desc>
    <path fill="#f26e7e" d="M4.683025 13.3318c0.869125 -0.5018 0.870575 -2.1264 0.003225 -3.62865s-2.27504 -2.313275 -3.1441725 -1.811475C0.672945 8.3935 0.6715 10.0181 1.53885 11.52035c0.86735 1.502275 2.27505 2.313275 3.144175 1.81145Zm0.0052 3.2167c0.86735 1.502275 0.865925 3.126875 -0.003225 3.628675 -0.86915 0.5018 -2.2768275 -0.309225 -3.144175 -1.81145 -0.8673525 -1.50225 -0.8659075 -3.126875 0.003225 -3.628675 0.8691325 -0.5018 2.276825 0.309225 3.144175 1.81145Zm5.922875 3.4243c0.86735 1.50225 0.8659 3.126775 -0.003225 3.62875 -0.869125 0.501775 -2.27685 -0.309325 -3.1442 -1.81155 -0.867325 -1.50225 -0.865875 -3.12685 0.00325 -3.628675 0.869125 -0.5018 2.276825 0.309225 3.144175 1.811475Zm-0.001925 -6.845275c0.86735 1.50225 0.8659 3.12685 -0.003225 3.628675 -0.869125 0.5018 -2.276825 -0.309225 -3.144175 -1.811475 -0.86735 -1.50225 -0.8659 -3.12685 0.003225 -3.62865 0.869125 -0.501825 2.276825 0.3092 3.144175 1.81145Z" stroke-width="0.25"></path>
    <path fill="#8a3391" d="M22.45775 11.524125c0.86725 -1.502225 0.865925 -3.12685 -0.003225 -3.62865 -0.869125 -0.501825 -2.276825 0.3092 -3.144175 1.811475 -0.86735 1.50225 -0.8659 3.126825 0.003225 3.62865 0.869125 0.501825 2.276825 -0.3092 3.144175 -1.811475Zm0.000175 3.2151c0.869075 0.5018 0.870625 2.1264 0.003225 3.62865 -0.86735 1.50225 -2.27505 2.313275 -3.144175 1.81145 -0.869125 -0.5018 -0.870575 -2.126425 -0.003225 -3.62865 0.86735 -1.50225 2.27505 -2.313275 3.144175 -1.81145ZM16.536225 18.157875c0.86915 0.501825 0.8706 2.126425 0.00325 3.628675 -0.86735 1.502125 -2.275075 2.313225 -3.1442 1.81145 -0.869125 -0.50175 -0.870575 -2.126425 -0.003225 -3.62865 0.867375 -1.502275 2.27505 -2.3133 3.144175 -1.811475Zm-0.003325 -6.843775c0.869125 0.5018 0.870575 2.126425 0.003225 3.628675s-2.27505 2.313275 -3.1442 1.811475c-0.869125 -0.501825 -0.870575 -2.126425 -0.003225 -3.628675 0.86735 -1.502275 2.27505 -2.313275 3.1442 -1.811475Z" stroke-width="0.25"></path>
    <path fill="#f7bf2a" d="M15.138225 2.06721c0 1.003615 -1.40625 1.817215 -3.14095 1.817215 -1.7347 0 -3.14095 -0.8136 -3.14095 -1.817215C8.856325 1.06359 10.262575 0.25 11.997275 0.25c1.7347 0 3.14095 0.81359 3.14095 1.81721ZM9.2166 5.482375c0 1.003625 -1.40625 1.8172 -3.14095 1.8172 -1.7347 0 -3.14095 -0.813575 -3.14095 -1.8172s1.40625 -1.817225 3.14095 -1.817225c1.7347 0 3.14095 0.8136 3.14095 1.817225Zm8.71005 1.8172c1.7347 0 3.14095 -0.813575 3.14095 -1.8172s-1.40625 -1.817225 -3.14095 -1.817225c-1.7347 0 -3.14095 0.8136 -3.14095 1.817225s1.40625 1.8172 3.14095 1.8172Zm-2.788425 1.605625c0 1.003625 -1.40625 1.8172 -3.14095 1.8172 -1.7347 0 -3.14095 -0.813575 -3.14095 -1.8172 0 -1.0036 1.40625 -1.8172 3.14095 -1.8172 1.7347 0 3.14095 0.8136 3.14095 1.8172Z" stroke-width="0.25"></path>
    </svg>}
  >
    Declare templates in your Pulumi programs
  </Card>
</CardGroup>

## Configuration

The default `values.yaml` for this template:

```yaml theme={null}
image: docker.elastic.co/elasticsearch/elasticsearch:8.17.0

replicas: 3 # Must be odd for master quorum (3, 5, 7, ...)

clusterName: my-elasticsearch-cluster

# JVM heap size per node — set to ~50% of maxMemory, hard cap at 30g
jvmHeap: 3g

resources:
  minCpu: 1
  minMemory: 2Gi
  maxCpu: 2
  maxMemory: 6Gi

volumeset:
  capacity: 10 # initial capacity in GiB (minimum is 10)
  autoscaling:
    enabled: false
    maxCapacity: 100
    minFreePercentage: 10
    scalingFactor: 1.2

multiZone:
  enabled: false # Set to true to schedule replicas across availability zones

internal_access:
  type: same-gvc # options: same-gvc, same-org, workload-list
  workloads:
    #- //gvc/GVC_NAME/workload/WORKLOAD_NAME

kibana:
  enabled: true
  image: docker.elastic.co/kibana/kibana:8.17.0
  resources:
    cpu: 500m
    memory: 2Gi
  internal_access:
    type: same-gvc # options: same-gvc, same-org, workload-list
    workloads:
      #- //gvc/GVC_NAME/workload/WORKLOAD_NAME

backup:
  enabled: false
  remove_setup_workload: false # Set to true after setup completes to reduce resource usage

  provider: aws # options: aws or gcp

  # Snapshot schedule in Quartz cron format (6 fields: seconds minutes hours day-of-month month day-of-week)
  schedule: "0 0 2 * * ?"

  retention:
    maxAge: 30d   # Delete snapshots older than this
    maxCount: 30  # Keep at most this many snapshots

  aws:
    bucket: my-s3-bucket
    region: us-east-1
    prefix: elasticsearch-snapshots
    cloudAccountName: my-cloud-account
    policyName: my-backup-policy

  gcp:
    bucket: my-gcs-bucket
    prefix: elasticsearch-snapshots
    cloudAccountName: my-cloud-account
```

### Cluster Settings

* `replicas` — Number of Elasticsearch nodes. **Must be an odd number** (3, 5, 7) to maintain a valid master election quorum.
* `clusterName` — The Elasticsearch cluster name. Used internally by nodes to discover each other.
* `jvmHeap` — JVM heap size per node. Set to approximately 50% of `maxMemory`. Hard cap at `30g`.

<Note>
  A minimum of 3 replicas is required for master quorum. With 3 nodes, the cluster can survive the loss of 1 node. Always use an odd number — an even number of nodes does not improve fault tolerance and can cause split-brain scenarios.
</Note>

### Resources and Storage

* `resources.minCpu` / `resources.minMemory` — Guaranteed minimum CPU and memory per node.
* `resources.maxCpu` / `resources.maxMemory` — Maximum CPU and memory per node.
* `volumeset.capacity` — Initial volume size in GiB per node (minimum 10).
* `volumeset.autoscaling.enabled` — Automatically expand volumes as data grows.
* `volumeset.autoscaling.maxCapacity` — Maximum volume size in GiB.
* `volumeset.autoscaling.minFreePercentage` — Triggers a scale-up when free space falls below this percentage.
* `volumeset.autoscaling.scalingFactor` — Multiplier applied to current capacity on each scale-up.

### Multi-Zone

When `multiZone.enabled: true`, Control Plane spreads Elasticsearch replicas across availability zones within the location. This improves durability — if a zone goes down, remaining nodes in other zones maintain quorum.

Verify your selected location supports multiple availability zones before enabling.

### Internal Access

Controls which workloads can reach Elasticsearch and Kibana. Both have independent `internal_access` settings.

| Type            | Description                                                     |
| --------------- | --------------------------------------------------------------- |
| `same-gvc`      | Allow access from all workloads in the same GVC (recommended)   |
| `same-org`      | Allow access from all workloads in the org                      |
| `workload-list` | Allow access only from specific workloads listed in `workloads` |

### Kibana

Kibana is enabled by default and provides a web UI for exploring data, managing indices, building dashboards, and monitoring cluster health.

* `kibana.enabled` — Deploy the Kibana workload.
* `kibana.resources.cpu` / `kibana.resources.memory` — CPU and memory for the Kibana container.
* `kibana.internal_access` — Controls which workloads can reach Kibana (same options as `internal_access`).

### Connecting

Both Elasticsearch and Kibana are accessible internally from within the same GVC. External access is blocked by default — use `cpln port-forward` to reach them from your local machine.

| Service       | Internal hostname                                    | Port   |
| ------------- | ---------------------------------------------------- | ------ |
| Elasticsearch | `{release-name}-elasticsearch.{gvc-name}.cpln.local` | `9200` |
| Kibana        | `{release-name}-kibana.{gvc-name}.cpln.local`        | `5601` |

**Port-forward to Kibana:**

```bash theme={null}
cpln workload port-forward --gvc GVC_NAME RELEASE_NAME-kibana --port 5601
# Then open http://localhost:5601 in your browser
```

**Port-forward to Elasticsearch:**

```bash theme={null}
cpln workload port-forward --gvc GVC_NAME RELEASE_NAME-elasticsearch --port 9200
```

### Scaling

**Scaling up** — Increase `replicas` to the next odd number and run `cpln helm upgrade`. New nodes join the cluster automatically and Elasticsearch begins rebalancing shards.

<Warning>
  **Scale down is destructive.** Elasticsearch does not automatically move shards off nodes that are about to be removed. Always take a manual snapshot before scaling down, and verify shard allocation with `GET /_cat/shards?v` before removing nodes to ensure no primary shards are stranded on the nodes being removed.
</Warning>

## Backup

Elasticsearch backups are **incremental snapshots** stored directly in S3 or GCS via the built-in repository-s3 and repository-gcs plugins — no separate backup image is required. The Snapshot Lifecycle Management (SLM) feature handles scheduling and retention automatically.

When backup is enabled, a one-time **Backup Setup Workload** runs at install time. It waits for the cluster to be healthy, then calls the Elasticsearch API to register the snapshot repository and create the SLM policy. Once it completes successfully, it can be removed to save resources:

```yaml theme={null}
backup:
  enabled: true
  remove_setup_workload: true
```

<Note>
  The backup schedule uses **Quartz cron format** with 6 fields (seconds, minutes, hours, day-of-month, month, day-of-week) — not the standard 5-field cron format. For example, `"0 0 2 * * ?"` runs daily at 2am UTC.
</Note>

### AWS S3 Prerequisites

1. Create an S3 bucket. Set `backup.aws.bucket` to its name and `backup.aws.region` to its region.
2. If you do not have a Cloud Account set up, refer to the docs to [Create a Cloud Account](/guides/create-cloud-account). Set `backup.aws.cloudAccountName` to its name.
3. Create an IAM policy with the following JSON, replacing `YOUR_BUCKET_NAME`:

```json theme={null}
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:ListBucket",
                "s3:GetObjectVersion",
                "s3:DeleteObjectVersion"
            ],
            "Resource": [
                "arn:aws:s3:::YOUR_BUCKET_NAME",
                "arn:aws:s3:::YOUR_BUCKET_NAME/*"
            ]
        }
    ]
}
```

4. Set `backup.aws.policyName` to the name of the policy created in step 3.

### GCS Prerequisites

1. Create a GCS bucket. Set `backup.gcp.bucket` to its name.
2. If you do not have a Cloud Account set up, refer to the docs to [Create a Cloud Account](/guides/create-cloud-account). Set `backup.gcp.cloudAccountName` to its name.
3. Add the **Storage Admin** role to the GCP service account associated with the Cloud Account.

### Manual Snapshots

Exec into any Elasticsearch container to trigger a snapshot immediately or inspect status:

```bash theme={null}
# Trigger a snapshot now via SLM
curl -X PUT 'http://localhost:9200/_slm/policy/automated-snapshots/_execute'

# List all snapshots in the repository
curl 'http://localhost:9200/_snapshot/backup-repo/_all?pretty'

# Check a snapshot currently in progress
curl 'http://localhost:9200/_snapshot/backup-repo/_current?pretty'

# View the SLM policy and last execution result
curl 'http://localhost:9200/_slm/policy/automated-snapshots?pretty'
```

## Restoring a Snapshot

Exec into any Elasticsearch node in the cluster to run restore commands. The snapshot repository (`backup-repo`) is already registered.

**List available snapshots:**

```bash theme={null}
curl 'http://localhost:9200/_snapshot/backup-repo/_all?pretty'
```

### Scenario 1 — Disaster Recovery (Fresh Cluster)

Deploy a new cluster from this template with backup enabled and pointing at the same bucket. Once the Backup Setup Workload completes (re-registering the same repository), restore all indices:

```bash theme={null}
curl -X POST 'http://localhost:9200/_snapshot/backup-repo/SNAPSHOT_NAME/_restore' \
  -H 'Content-Type: application/json' \
  -d '{
    "indices": ["*", "-.internal.*", "-.slo-observability.*.temp", "-.ds-ilm-history*"],
    "ignore_unavailable": true,
    "include_global_state": false
  }'
```

### Scenario 2 — Restore to Existing Cluster

When the target indices already exist, close them before restoring or the restore will be rejected:

```bash theme={null}
# Close the index first
curl -X POST 'http://localhost:9200/MY_INDEX/_close'

# Restore from snapshot
curl -X POST 'http://localhost:9200/_snapshot/backup-repo/SNAPSHOT_NAME/_restore' \
  -H 'Content-Type: application/json' \
  -d '{
    "indices": "MY_INDEX",
    "ignore_unavailable": true,
    "include_global_state": false
  }'
```

### Scenario 3 — Restore Specific Indices

```bash theme={null}
curl -X POST 'http://localhost:9200/_snapshot/backup-repo/SNAPSHOT_NAME/_restore' \
  -H 'Content-Type: application/json' \
  -d '{
    "indices": "my-index,my-other-index-2026.05*",
    "ignore_unavailable": true,
    "include_global_state": false
  }'
```

**Monitor restore progress:**

```bash theme={null}
# View active recovery operations
curl 'http://localhost:9200/_cat/recovery?v&active_only=true'

# Check overall cluster health
curl 'http://localhost:9200/_cluster/health?pretty'
```

## Important Notes

* **Replica count must be odd** — Elasticsearch requires an odd number of master-eligible nodes for quorum. Even numbers do not improve fault tolerance and can cause split-brain.
* **JVM heap** — Set `jvmHeap` to approximately 50% of `maxMemory`, with a hard cap of `30g`. Elasticsearch relies heavily on off-heap memory for the filesystem cache.
* **Backup schedule format** — SLM uses Quartz cron format (6 fields), not the standard 5-field cron. The `?` wildcard is required in the day-of-week field when day-of-month is set.
* **Scale down carefully** — Always snapshot before reducing replicas. Verify no primary shards exist on nodes being removed before running the upgrade.

## External References

<CardGroup cols={2}>
  <Card title="Elasticsearch Documentation" icon="book" href="https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html">
    Official Elasticsearch reference documentation
  </Card>

  <Card title="Kibana Documentation" icon="book" href="https://www.elastic.co/guide/en/kibana/current/index.html">
    Official Kibana guide
  </Card>

  <Card title="Snapshot and Restore" icon="book" href="https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshot-restore.html">
    Elasticsearch snapshot and restore guide
  </Card>

  <Card title="Snapshot Lifecycle Management" icon="book" href="https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshots-take-snapshot.html">
    Create, monitor, and delete snapshots with SLM
  </Card>

  <Card title="Elasticsearch Template" icon="github" href="https://github.com/controlplane-com/templates/tree/main/elasticsearch">
    View the source files, default values, and chart definition
  </Card>
</CardGroup>