OpenSearch

Overview

OpenSearch is an open-source distributed search and analytics engine for logs, metrics, and full-text search. This template deploys a multi-node OpenSearch cluster with persistent storage per node, an OpenSearch Dashboards UI (on by default), an optional demo log pipeline that ships sample logs into the cluster, and optional scheduled snapshots to AWS S3 or Google Cloud Storage.

This template deploys into an existing GVC that you already have. It does not create one.

What Gets Created

Resource	Name	Purpose	Optional
Workload	`RELEASE_NAME-opensearch`	OpenSearch nodes (stateful, `replicas` replicas); HTTP on `9200`, transport on `9300`	—
Volume Set	`RELEASE_NAME-opensearch-vs`	Persistent data directory per node, with optional autoscaling	—
Secret	`RELEASE_NAME-opensearch-startup`	Node startup script; installs the snapshot repository plugin when backups are enabled	—
Identity	`RELEASE_NAME-opensearch-identity`	Identity for the OpenSearch and Dashboards workloads; carries the cloud account binding when backups are enabled	—
Policy	`RELEASE_NAME-opensearch-policy`	Grants the identity `reveal` on the startup secret	—
Workload	`RELEASE_NAME-opensearch-dashboard`	OpenSearch Dashboards UI on port `5601`	`dashboard.enabled: true` (default)
Workload	`RELEASE_NAME-backup-setup`	One-time job that registers the `backup-repo` snapshot repository and the snapshot policy	`backup.enabled: true`
Workload	`RELEASE_NAME-demo-log-generator`	Sample log generator with a Fluent Bit sidecar shipping into the `demo-logs` index	`demoLogs.enabled: true`
Workload	`RELEASE_NAME-demo-log-setup`	One-time job that creates the demo index template and the `demo-logs*` index pattern in Dashboards	`demoLogs.enabled: true`
Volume Set	`RELEASE_NAME-demo-logs-vs`	Shared log directory between the generator and Fluent Bit	`demoLogs.enabled: true`
Secret	`RELEASE_NAME-fluent-bit-demo-config`	Fluent Bit configuration for the demo pipeline	`demoLogs.enabled: true`
Identity	`RELEASE_NAME-demo-logs-identity`	Identity for the demo pipeline workload	`demoLogs.enabled: true`
Policy	`RELEASE_NAME-demo-logs-policy`	Grants the demo identity `reveal` on the Fluent Bit config secret only	`demoLogs.enabled: true`

Prerequisites

None for a default install. Scheduled snapshots need a bucket and a Control Plane Cloud Account before backup.enabled can be turned on. See Backing Up for the per-provider steps.

Installation

Install the latest version into an existing GVC. A default install has no required --set values:

cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/opensearch \
  --version 1.0.2 --gvc GVC_NAME

To install using another method, follow the instructions for your preferred tool:

UI

Browse, install, and manage templates visually

CLI

Manage templates from your terminal

Terraform

Declare templates in your Terraform configurations

Pulumi

Declare templates in your Pulumi programs

Configuration

Each subsection shows the shipped defaults for one area of values.yaml.

Cluster

replicas must be odd so the cluster can form a quorum; an even value is rejected when the chart renders.

image: opensearchproject/opensearch:3.4.0

replicas: 3 # Must be odd

clusterName: my-opensearch-cluster

Resources

Per-node limits and reservations. OpenSearch is memory-sensitive; raise maxMemory before raising CPU.

resources:
  minCpu: 500m
  minMemory: 2Gi
  maxCpu: 1
  maxMemory: 4Gi

Volume Set

One volume per node. Enable autoscaling to let each volume grow as indices fill it.

volumeset:
  capacity: 10 # initial capacity in GiB (minimum is 10)
  autoscaling:
    enabled: false # Set to true to enable autoscaling
    maxCapacity: 100 # Maximum capacity in GiB when autoscaling is enabled
    minFreePercentage: 10 # Minimum free percentage to trigger scaling when autoscaling is enabled
    scalingFactor: 1.2 # Scaling factor to determine how much to scale up when autoscaling is triggered

Internal Access

Controls which workloads may reach OpenSearch on port 9200. External inbound access is closed and is not configurable in this template. Because the security plugin is disabled, this setting is the only access control the cluster has.

internal_access:
  type: same-gvc # options: same-gvc, same-org, workload-list
  workloads:  # Note: can only be used if type is same-gvc or workload-list
    #- //gvc/GVC_NAME/workload/WORKLOAD_NAME
    #- //gvc/GVC_NAME/workload/WORKLOAD_NAME

Type	Who can connect
`same-gvc`	Every workload in the same GVC
`same-org`	Every workload in the same org
`workload-list`	Only the workloads listed under `workloads`

Dashboards

The OpenSearch Dashboards UI. It is not exposed to the internet; reach it with cpln port-forward as shown in Connecting.

# Dashboard Configuration (recommended)
dashboard:
  enabled: true
  image: opensearchproject/opensearch-dashboards:3.4.0
  resources:
    cpu: 100m
    memory: 512Mi

Demo Logs

A sample log generator with a Fluent Bit sidecar that ships JSON logs into a demo-logs index, plus a one-time setup job that creates the demo-logs-template index template and makes demo-logs* the default index pattern in Dashboards. The setup job needs dashboard.enabled: true.

# Demo Logs Configuration (optional)
demoLogs:
  enabled: false
  remove_setup_workload: false # Set to true after demo logs startup is no longer needed to reduce resource usage

Backup

Scheduled snapshots to AWS S3 or GCS. Complete the provider steps in Backing Up before setting enabled: true. When provider is aws, aws.bucket, aws.region, aws.cloudAccountName and aws.policyName are required; when it is gcp, gcp.bucket and gcp.cloudAccountName are required. Any other provider value is rejected when the chart renders.

# Backup Configuration (optional)
backup:
  enabled: false      # Set to true to enable automated snapshots
  remove_setup_workload: false # Set to true after backup setup is no longer needed to reduce resource usage

  # Cloud provider: aws or gcp
  provider: aws

  # Backup schedule (cron format, UTC timezone)
  schedule: 0 2 * * *  # Daily at 2am UTC

  # Snapshot retention
  retention:
    maxAge: 30d        # Delete snapshots older than 30 days
    maxCount: 30       # Keep maximum 30 snapshots

  # AWS S3 Configuration
  aws:
    bucket: my-s3-bucket                 # S3 bucket name (required if provider=aws)
    region: us-east-1                    # S3 bucket region (required if provider=aws)
    prefix: opensearch-snapshots         # Path prefix in bucket
    cloudAccountName: my-cloud-account   # Control Plane Cloud Account name (required)
    policyName: my-backup-policy         # AWS IAM custom policy name (required)

  # GCP GCS Configuration
  gcp:
    bucket: my-gcs-bucket                # GCS bucket name (required if provider=gcp)
    prefix: opensearch-snapshots         # Path prefix in bucket
    cloudAccountName: my-cloud-account   # Control Plane Cloud Account name (required)

Connecting

Neither workload has a public endpoint. Applications connect over the internal network; you reach the API and the UI from your machine with cpln port-forward.

What	Address	Notes
OpenSearch REST API (from workloads in the GVC)	`http://RELEASE_NAME-opensearch.GVC_NAME.cpln.local:9200`	Always use the fully-qualified hostname; subject to `internal_access`
OpenSearch REST API (from your machine)	`cpln port-forward RELEASE_NAME-opensearch 9200:9200 --gvc GVC_NAME`, then `http://localhost:9200`	Works regardless of `internal_access`
Dashboards UI	`cpln port-forward RELEASE_NAME-opensearch-dashboard 5601:5601 --gvc GVC_NAME`, then `http://localhost:5601`	Only when `dashboard.enabled: true`
Credentials	None	The security plugin is disabled; there is no username or password

Verify the cluster is healthy and all nodes have joined:

cpln port-forward RELEASE_NAME-opensearch 9200:9200 --gvc GVC_NAME

In a second terminal:

curl "http://localhost:9200/_cluster/health?pretty"

A healthy default install reports "status" : "green" and "number_of_nodes" : 3.

Operations

Backing Up

The template takes snapshots with OpenSearch’s own snapshot API. When backup.enabled: true, each node installs the repository-s3 or repository-gcs plugin at startup, and the RELEASE_NAME-backup-setup job registers a snapshot repository named backup-repo plus a snapshot management policy named automated-snapshots that runs on backup.schedule and prunes snapshots daily according to backup.retention. Snapshots cover all indices ("indices": "*") and exclude cluster global state.

Enable backups at install time if you can. Turning them on later works, but every node has to restart to install the repository plugin before the repository can be registered — see Troubleshooting.

Set up your provider first.

AWS S3
Google Cloud Storage

Create the bucket

Create an S3 bucket. Set backup.aws.bucket to its name and backup.aws.region to its region.

Create or choose a Cloud Account

If you do not have one, follow Create a Cloud Account. Set backup.aws.cloudAccountName to its name.

Create the IAM policy

Create an IAM policy with the following JSON, replacing YOUR_BUCKET_NAME. Set backup.aws.policyName to the policy’s name.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:ListBucket",
                "s3:GetBucketLocation",
                "s3:ListBucketMultipartUploads",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts"
            ],
            "Resource": [
                "arn:aws:s3:::YOUR_BUCKET_NAME",
                "arn:aws:s3:::YOUR_BUCKET_NAME/*"
            ]
        }
    ]
}

Both Resource ARNs are required. s3:ListBucket and s3:GetBucketLocation authorize against the bucket itself, while the object actions authorize against YOUR_BUCKET_NAME/*. A policy carrying only the /* ARN registers the repository and then fails when OpenSearch lists it.

Enable backups

Set backup.enabled: true and backup.provider: aws, then install or upgrade the release.

Create the bucket

Create a GCS bucket. Set backup.gcp.bucket to its name.

Create or choose a Cloud Account

If you do not have one, follow Create a Cloud Account. Set backup.gcp.cloudAccountName to its name.

Grant storage access

Add the Storage Admin role to the GCP service account associated with the Cloud Account. Access is keyless; no credentials are stored in the template.

Enable backups

Set backup.enabled: true and backup.provider: gcp, then install or upgrade the release.

Confirm the setup job finished by reading its logs; it ends with Backup Configuration Complete!:

cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-backup-setup"}' --limit 50 --since 10m

Once it has completed, set backup.remove_setup_workload: true and upgrade the release to remove the setup workload. The repository and policy live inside OpenSearch, so scheduled snapshots continue without it. With a port-forward to RELEASE_NAME-opensearch open on 9200 (see Connecting), take or inspect snapshots from your machine:

# Take a manual snapshot
curl -X PUT "http://localhost:9200/_snapshot/backup-repo/manual-$(date +%Y%m%d-%H%M%S)"

# List all snapshots in the repository
curl "http://localhost:9200/_snapshot/backup-repo/_all?pretty"

# Show the automated snapshot policy
curl "http://localhost:9200/_plugins/_sm/policies/automated-snapshots?pretty"

Restoring a Backup

OpenSearch snapshots are restored through the snapshot API, from the same backup-repo repository the template registers. The commands below assume a port-forward to RELEASE_NAME-opensearch on 9200 (see Connecting).

These restore commands follow the OpenSearch snapshot API against the repository this template configures, but the template’s test suite has not exercised a snapshot round trip against a live bucket. Rehearse a restore in a non-production cluster before you depend on it.

Into an empty cluster — nothing to conflict with:

curl -X POST "http://localhost:9200/_snapshot/backup-repo/SNAPSHOT_NAME/_restore" \
  -H 'Content-Type: application/json' \
  -d '{"indices": "*", "ignore_unavailable": true, "include_global_state": false}'

Alongside existing indices — restore under new names and leave live data untouched. This is the safest option on a cluster that is in use:

curl -X POST "http://localhost:9200/_snapshot/backup-repo/SNAPSHOT_NAME/_restore" \
  -H 'Content-Type: application/json' \
  -d '{"indices": "*", "rename_pattern": "(.+)", "rename_replacement": "restored-$1",
       "ignore_unavailable": true, "include_global_state": false}'

Over an existing index — an open index cannot be restored into, so close it first, restore just that index, and reopen it:

curl -X POST "http://localhost:9200/INDEX_NAME/_close"
curl -X POST "http://localhost:9200/_snapshot/backup-repo/SNAPSHOT_NAME/_restore" \
  -H 'Content-Type: application/json' \
  -d '{"indices": "INDEX_NAME", "ignore_unavailable": true, "include_global_state": false}'
curl -X POST "http://localhost:9200/INDEX_NAME/_open"

Watch the restore progress and wait for the cluster to return to green:

curl "http://localhost:9200/_cat/recovery?v&active_only=true"
curl "http://localhost:9200/_cluster/health?pretty"

Upgrading from 1.0.x

Version 1.0.2 fixes five defects, all present since 1.0.0. Two of them matter regardless of how you configured the template. Upgrade with cpln helm upgrade against version 1.0.2 using your existing values; the data on the volume sets is retained. The identity links were hardcoded to test-gvc. Both workloads’ identityLink and both policies’ principalLinks pointed at //gvc/test-gvc/identity/... rather than at your GVC, so an install of 1.0.0 or 1.0.1 into any GVC not literally named test-gvc could not start. There is nothing to migrate: upgrade to 1.0.2 and the links resolve correctly. The demo-logs policy granted reveal on every secret in your org. When demoLogs.enabled: true, the demo-logs policy carried a query that matched every secret in the organization alongside the one Fluent Bit config secret it needed, so the demo-logs identity could read all of them.

If you ran 1.0.0 or 1.0.1 with demoLogs.enabled: true, treat every secret in that organization as having been readable by that identity. Upgrading to 1.0.2 narrows the policy to the single Fluent Bit config secret, but it does not undo past access. Review what that organization holds and rotate anything sensitive.

The remaining three:

A GCP backup also emitted an AWS identity binding pointing at a nonexistent cloud account. The provider conditions now render only the binding for the selected provider.
aws::ReadOnlyAccess is no longer attached to the identity. It granted read on every bucket in the account and carried none of the write actions a snapshot repository needs; backups work without it.
An unsupported backup.provider and an even replicas count now fail when the chart renders instead of installing cleanly and misbehaving later.

Scaling and Availability

Cluster size. replicas sets the number of OpenSearch nodes and must be odd. Three nodes is the smallest size that tolerates the loss of one node; use five or seven for larger clusters.
Per-node resources. resources.maxMemory is the first knob to raise as index volume grows; keep minCpu/maxCpu within the same proportions as the defaults.
Storage. Set volumeset.capacity for the data you expect, or enable volumeset.autoscaling so each node’s volume grows as it fills. Volumes cannot shrink.
Resizing a running cluster was not exercised by this template’s tests. Take a snapshot first, and follow the OpenSearch cluster documentation for adding or removing nodes safely.

Troubleshooting

Backup setup logs 'repository type [s3] does not exist' (or [gcs]) and restarts

Cause: backup.enabled was turned on for a cluster that was already running. The repository plugin is installed by each node’s startup script, so the repository cannot be registered until every node has restarted with the plugin present. The setup job retries until that happens.Fix: Wait. The job converges on its own once all nodes carry the plugin, and later runs skip the steps already completed. Check progress with:

cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-backup-setup"}' --limit 50 --since 10m

Enabling backups at install time avoids this entirely.

Repository registers, but listing or taking snapshots fails with an S3 access-denied error

Cause: The IAM policy grants object actions on arn:aws:s3:::YOUR_BUCKET_NAME/* only. s3:ListBucket and s3:GetBucketLocation authorize against the bucket ARN itself, so registration succeeds and the first operation that enumerates the repository is refused.Fix: Add arn:aws:s3:::YOUR_BUCKET_NAME to the policy’s Resource list, as shown in Backing Up.

A 1.0.0 or 1.0.1 install never becomes ready

Cause: Those versions referenced an identity in a GVC named test-gvc instead of the GVC you installed into, so the workloads could not start anywhere else.Fix: Upgrade to 1.0.2. See Upgrading from 1.0.x.

Applications report 'could not resolve host' for the OpenSearch workload

Cause: The application is using the bare workload name. The short name is not reliably resolvable.Fix: Use the fully-qualified hostname RELEASE_NAME-opensearch.GVC_NAME.cpln.local:9200, and confirm the calling workload is admitted by internal_access.

The demo setup job waits on 'Waiting for Dashboard...' and eventually exits

Cause: demoLogs.enabled: true with dashboard.enabled: false. The demo setup job creates the demo-logs* index pattern in Dashboards and cannot complete without it.Fix: Set dashboard.enabled: true and upgrade the release, or disable the demo pipeline.

Important Notes

There is no authentication. The security plugin is disabled, so any workload admitted by internal_access has full read/write admin access to every index. Keep internal_access.type as narrow as you can, and prefer workload-list over same-org for anything sensitive.
replicas must be odd. An even count cannot form a quorum and is rejected when the chart renders.
Still on 1.0.0 or 1.0.1? Upgrade. Both versions are broken outside a GVC named test-gvc, and the demo pipeline granted org-wide secret access. See Upgrading from 1.0.x.
Enable backups at install time when you can; enabling them on a running cluster requires every node to restart before the repository can be registered.
Switching backup.provider on an existing release leaves the previous provider’s cloud binding attached to the identity. To remove it you must uninstall and reinstall; take a snapshot first, because uninstall deletes the volume sets.
Use the fully-qualified internal hostname RELEASE_NAME-opensearch.GVC_NAME.cpln.local:9200; the bare workload name is not reliably resolvable.
cpln helm uninstall deletes the volume sets. Data survives a cpln helm upgrade but not an uninstall; enable snapshots to a bucket if the data matters.

External References

OpenSearch Documentation

Official OpenSearch documentation

OpenSearch Dashboards

OpenSearch Dashboards documentation

Snapshot and Restore

Taking and restoring snapshots through the API

Snapshot Management

Automated snapshot policies and retention

OpenSearch Template

View the source files, default values, and chart definition

Template Catalog

Templates

Overview

What Gets Created

Prerequisites

Installation

UI

CLI

Terraform

Pulumi

Configuration

Cluster

Resources

Volume Set

Internal Access

Dashboards

Demo Logs

Backup

Connecting

Operations

Backing Up

Restoring a Backup

Upgrading from 1.0.x

Scaling and Availability

Troubleshooting

Important Notes

External References

OpenSearch Documentation

OpenSearch Dashboards

Snapshot and Restore

Snapshot Management

OpenSearch Template

​Overview

​What Gets Created

​Prerequisites

​Installation

UI

CLI

Terraform

Pulumi

​Configuration

​Cluster

​Resources

​Volume Set

​Internal Access

​Dashboards

​Demo Logs

​Backup

​Connecting

​Operations

​Backing Up

​Restoring a Backup

​Upgrading from 1.0.x

​Scaling and Availability

​Troubleshooting

​Important Notes

​External References

OpenSearch Documentation

OpenSearch Dashboards

Snapshot and Restore

Snapshot Management

OpenSearch Template

Overview

What Gets Created

Prerequisites

Installation

Configuration

Cluster

Resources

Volume Set

Internal Access

Dashboards

Demo Logs

Backup

Connecting

Operations

Backing Up

Restoring a Backup

Upgrading from 1.0.x

Scaling and Availability

Troubleshooting

Important Notes

External References