Manticore Search

Overview

Upgrading from 2.0.1 or earlier? Two values keys were removed and one default changed — see Upgrading From 2.0.1 or Earlier. Anyone on an earlier version should also treat their agent token as compromised: it shipped as a working default in this repository and was shared by every install that did not override it.

Manticore Search is a high-performance, open-source search engine built for fast full-text search at scale. This template deploys a distributed Manticore Search cluster using Galera replication for high availability, with an orchestrator API for cluster management, a web UI for monitoring and operations, and support for zero-downtime data imports from AWS S3.

What Gets Created

Manticore Search Workload — A stateful Galera cluster with configurable replicas. Each replica runs a sidecar agent that handles cluster coordination, data imports, and recovery operations.
Orchestrator API Workload — A REST API service for triggering imports, monitoring cluster health, initiating repairs, and managing backup and restore operations.
Orchestrator Cron Job — A scheduled workload that runs import, health check, or repair operations on a configured schedule.
UI Workload — A web dashboard for managing the cluster, monitoring replication, triggering operations, and visualizing queries.
Volume Set — Persistent storage allocated per replica for Manticore data directories.
Shared Volume Set — A shared volume accessible by all replicas and the orchestrator, used for slot-based import coordination.
Secrets — Four opaque secrets: the Manticore searchd base configuration, the startup/shutdown handler script, the table schema registry used by the agent, and the agent bearer token. A K6 load test script secret is also created when load testing is enabled.
Identities & Policies — Identities for the main cluster, orchestrator, and backup workloads. Policies grant reveal access to the configuration secrets, exec permissions on the orchestrator cron job, and view access to the Manticore workload for orchestration.
Backup Cron Job (optional) — Scheduled logical backups of delta and full tables to an S3 bucket. Enabled when orchestrator.backup.enabled: true.
Domain (optional) — Routes /api/* to the orchestrator API and all other traffic to the UI. Enabled when domain.enabled: true.
Load Test Workload (optional) — A k6-based load test runner with a controller for automated scheduling. Enabled when loadTest.enabled: true.

This template does not create a GVC. You must deploy it into an existing GVC.

Prerequisites

This template requires an AWS S3 bucket for CSV source data and a Control Plane Cloud Account before installation.

AWS S3 (Source Data)

Create an S3 bucket and upload your CSV source files. Set buckets.sourceBucket to the bucket name and buckets.awsRegion to its region.
If you do not have a Control Plane Cloud Account set up, follow the Create a Cloud Account guide. Set buckets.cloudAccountName to the name of your Cloud Account.
Set buckets.awsPolicyRefs to the IAM policies granting S3 access. Read-only is sufficient — use the AWS managed policy aws::AmazonS3ReadOnlyAccess, or a custom policy with s3:GetObject and s3:ListBucket on the bucket (omit the aws:: prefix for custom policies).

Versions before 2.1.0 documented s3:PutObject and s3:DeleteObject on this bucket. That was never required: imports read from S3 and write only to the shared volume. If you granted write access on your source data bucket, you can narrow it to read-only. Backups use a separate bucket, which does need write access.

Agent Token

The orchestrator, the agent sidecars and the UI authenticate to each other with a single bearer token. It is a prerequisite opaque secret that must exist before you install — the deployment otherwise hangs waiting on a secret that is not there, which looks like a platform fault rather than a missing step.

printf '%s' "$(openssl rand -base64 32)" | \
  cpln secret create-opaque --name my-manticore-agent-token --encoding plain -f -

Set orchestrator.agent.tokenSecretName to that name.

The management UI has no authentication of its own. It holds the bearer token server-side and injects it on behalf of whoever connects, so the token protects the orchestrator API but not the UI itself. Anyone who can reach the UI has full cluster admin — imports, repairs and cluster operations.This is why orchestrator.ui.publicAccess.enabled defaults to false. Turning it on publishes cluster admin to the internet. If you need browser access from outside the GVC, put your own authenticating proxy in front of it.

AWS S3 (Backup, Optional)

Only required if orchestrator.backup.enabled: true:

Create a separate S3 bucket for backups. Set orchestrator.backup.s3Bucket and orchestrator.backup.s3Region.
Create an IAM policy with the following permissions, replacing YOUR_BUCKET_NAME:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::YOUR_BUCKET_NAME",
                "arn:aws:s3:::YOUR_BUCKET_NAME/*"
            ]
        }
    ]
}

Set orchestrator.backup.cloudAccountName and orchestrator.backup.s3Policy to the name of your Cloud Account and the custom policy created above.

Installation

To install, follow the instructions for your preferred method:

UI

Browse, install, and manage templates visually

CLI

Manage templates from your terminal

Terraform

Declare templates in your Terraform configurations

Pulumi

Declare templates in your Pulumi programs

Configuration

The default values.yaml for this template:

# =============================================================================
# AWS Cloud Account and S3 Configuration
# =============================================================================
buckets:
  cloudAccountName: my-manticore-cloudaccount    # Name of your configured Cloud Account
  awsPolicyRefs:                        # IAM policies for S3 access
    - aws::AmazonS3ReadOnlyAccess       # Note: if using a custom policy, omit the aws:: prefix
  awsRegion: us-east-1                  # Region of your S3 bucket
  sourceBucket: my-manticore-bucket               # S3 bucket containing files to import

# =============================================================================
# Tables Configuration
# =============================================================================
tables:
  - name: addresses
    csvPath:
      - manticore-imports/addresses/data.csv
    config:
      haStrategy: noerrors
      agentRetryCount: 3
      clusterMain: false
      segmentCount: 1
      charsetTable: non_cont
      memLimit: 2G
      hasHeader: true
      secondaryIndexes: false
    schema:
      columns:
        - name: address_id
          type: attr_uint
        - name: street_number
          type: attr_uint
        - name: street_name
          type: field
        - name: city
          type: field
        - name: county
          type: field
        - name: state
          type: field
        - name: postal_code
          type: field
        - name: country
          type: field
        - name: latitude
          type: attr_float
        - name: longitude
          type: attr_float

# =============================================================================
# Manticore Search Configuration
# =============================================================================
manticore:
  image: manticoresearch/manticore:25.0.0
  clusterName: manticore          # Galera cluster name
  resources:
    cpu: 4
    memory: 8Gi
  volumeset:
    capacity: 200                 # GB per replica
  sharedVolumeset:
    capacity: 100                 # GB shared across replicas and orchestrator

  # minScale = replica count used by orchestrator for coordination
  autoscaling:
    minScale: 3
    maxScale: 4
    metric: rps                   # rps for stateful, cpu for standard
    target: 100
    scaleToZeroDelay: 300

  rolloutOptions:
    maxSurgeReplicas: 25%
    minReadySeconds: 10
    scalingPolicy: OrderedReady
    terminationGracePeriodSeconds: 60

  # IMPORTANT: same-gvc required for Galera replication
  firewall:
    internalAccess:
      type: same-gvc
      workloads: []

# =============================================================================
# Orchestrator Configuration
# =============================================================================
orchestrator:
  version: v6.0.5
  image: ghcr.io/controlplane-com/manticore-orchestrator/manticore-cpln-api
  logLevel: debug                 # debug, info, warn, error
  resources:
    cpu: 1
    memory: 2Gi

  # Cron job settings
  schedule: "0 * * * *"           # Cron schedule (default = every hour)
  action: import                  # init, import, health, repair
  tableName: addresses            # Must match a name in tables[]
  suspend: true                   # Start suspended (trigger via UI/API)
  timeoutSeconds: 900             # Container timeout (default 15 minutes)
  importMemLimit: 2G              # Memory limit for import jobs
  activeDeadlineSeconds: 14400    # Max job runtime (default 4 hours)

  # Orchestrator API
  api:
    version: v6.0.5
    image: ghcr.io/controlplane-com/manticore-orchestrator/manticore-cpln-api
    logLevel: debug
    importPollInterval: 30s
    importPollTimeout: 2h
    resources:
      cpu: 0.25
      memory: 256Mi
    autoscaling:
      maxScale: 3
      minScale: 2
      metric: cpu
      target: 80

  # Agent sidecar
  agent:
    version: v6.0.5
    image: ghcr.io/controlplane-com/manticore-orchestrator/manticore-cpln-agent
    # REQUIRED prerequisite: name of an opaque secret holding the bearer token.
    # THE SECRET MUST EXIST BEFORE INSTALL — without it the deploy hangs waiting on a missing secret.
    # printf '%s' "$(openssl rand -base64 32)" | cpln secret create-opaque --name my-manticore-agent-token --encoding plain -f -
    tokenSecretName: my-manticore-agent-token
    resources:
      cpu: 250m
      minCpu: 100m
      memory: 512Mi
      minMemory: 128Mi
    import:
      batchSize: 20000            # Rows per INSERT statement
    recovery:
      maxRetries: 5               # Retry attempts for cluster recovery
      initialBackoffSec: 5        # Initial delay between retries
      maxBackoffSec: 60           # Max backoff delay (exponential)

  # Web UI
  ui:
    version: v6.0.5
    image: ghcr.io/controlplane-com/manticore-orchestrator/manticore-cpln-ui
    resources:
      cpu: 0.25
      memory: 0.25Gi
    # DANGER: the UI has NO login of its own and injects the admin token for every caller.
    # Enabling this publishes full cluster admin to the internet. Put an authenticating proxy in front.
    publicAccess:
      enabled: false
    internalAccess:
      type: same-gvc              # same-gvc, same-org, workload-list, none
      workloads: []               # //gvc/{gvc}/workload/{name} links, used when type is workload-list
    autoscaling:
      maxScale: 2
      minScale: 1
      metric: cpu
      target: 80

  # Backup Configuration (Optional)
  backup:
    enabled: false
    version: v6.0.5
    image: ghcr.io/controlplane-com/manticore-orchestrator/manticore-cpln-backup
    cloudAccountName: my-manticore-backup-cloudaccount
    s3Bucket: my-manticore-backup-bucket
    s3Policy:
      - my-backup-policy
    s3Region: us-east-1
    dataSet: addresses            # Data set to back up
    prefix: manticore-backups     # S3 prefix/folder for backups
    schedules:
      - {"table":"addresses","type":"delta","schedule":"0 2 * * *"}   # Daily at 2am UTC
      - {"table":"addresses","type":"main","schedule":"0 2 1 * *"}    # Monthly on 1st at 2am UTC
    activeDeadlineSeconds: 14400
    resources:
      cpu: 1
      memory: 1Gi

# =============================================================================
# Domain Configuration (Optional)
# =============================================================================
domain:
  enabled: false
  name: ""                        # FQDN, e.g., manticore.example.com
  dnsMode: cname                  # cname (subdomains) or ns (zone delegation)

# =============================================================================
# Load Testing Configuration (Optional)
# =============================================================================
loadTest:
  enabled: false
  image: grafana/k6:0.47.0
  resources:
    cpu: 0.5
    memory: 512Mi
  vus: 10                         # Virtual users
  duration: "5m"                  # Test duration (e.g., 30s, 5m, 1h)
  rps: null                       # Target RPS (null = unlimited)
  replicas: 1
  controller:
    image: alpine/curl
    schedule: ""                  # Cron expression (empty = manual only)
    testDurationBuffer: 60        # Seconds added to duration before scale-down
  target:
    port: 9308                    # Manticore HTTP API port
    endpoint: search              # "search" or "sql"
  query:
    index: addresses
    query:
      match:
        "*": "test"
    limit: 10
  thresholds:
    p95ResponseTime: 500          # ms
    errorRate: 0.01               # 1%

S3 and Cloud Account

buckets.cloudAccountName — Name of the Control Plane Cloud Account with AWS trust configured.
buckets.awsPolicyRefs — IAM policies granting read access to the source S3 bucket. Use aws::AmazonS3ReadOnlyAccess for the AWS managed policy, or omit the aws:: prefix for custom policies.
buckets.awsRegion — AWS region of the source S3 bucket.
buckets.sourceBucket — Name of the S3 bucket containing CSV files to import.

Tables

Each entry in tables defines a searchable index imported from a CSV file.

name — Table name used by Manticore and referenced by the orchestrator.
csvPath — List of S3 paths (relative to sourceBucket) for the CSV source files. Multiple paths create a distributed multi-segment table.
config.haStrategy — High-availability behavior when agents are unreachable (noerrors ignores unreachable agents).
config.agentRetryCount — Number of retry attempts when an agent doesn’t respond.
config.segmentCount — Number of distributed segments for this table. Use more than 1 for very large datasets split across multiple CSVs.
config.memLimit — Memory limit for the Manticore index (e.g., 2G).
config.hasHeader — Set to true if the CSV file includes a header row.
config.secondaryIndexes — When true, enables secondary indexes on both the main and delta tables for this entry. Defaults to false.
schema.columns — Column definitions. Each column has a name and a type.

Supported column types:

Type	Description
`field`	Full-text searchable string field
`attr_uint`	Unsigned integer attribute
`attr_float`	Float attribute
`attr_bigint`	64-bit integer attribute
`attr_bool`	Boolean attribute
`attr_string`	Non-indexed string attribute
`attr_json`	JSON attribute
`attr_multi`	Multi-value unsigned integer attribute
`attr_multi_64`	Multi-value 64-bit integer attribute
`attr_timestamp`	Unix timestamp attribute

Manticore Cluster

manticore.clusterName — Galera cluster name used for replication coordination.
manticore.resources — CPU and memory for each Manticore replica.
manticore.volumeset.capacity — Persistent storage per replica in GB.
manticore.sharedVolumeset.capacity — Shared storage in GB, accessible by all replicas and the orchestrator for import slot coordination.
manticore.autoscaling.minScale — Minimum replica count. This value is also used by the orchestrator to determine cluster quorum. It should match your intended fixed replica count.
manticore.autoscaling.maxScale — Maximum replica count for autoscaling.

manticore.firewall.internalAccess.type must be set to same-gvc. Galera replication requires direct peer-to-peer communication between all replicas within the GVC.

Orchestrator

The orchestrator manages all cluster lifecycle operations including data imports, health checks, and repairs. Cron Job

orchestrator.schedule — Cron schedule for automated operations (default: every hour).
orchestrator.action — Operation to run: init (initial cluster setup), import (load CSV data), health (check cluster state), or repair (fix split-brain issues).
orchestrator.tableName — The table to target for the cron job. Must match a name in tables.
orchestrator.suspend — When true, the cron job is created in a suspended state and must be triggered manually via the UI or CLI. Recommended for production.
orchestrator.timeoutSeconds — Timeout for each cron container execution.
orchestrator.activeDeadlineSeconds — Maximum total runtime for an import job before it is terminated.

API

orchestrator.api.importPollInterval — How often the API checks import progress.
orchestrator.api.importPollTimeout — Maximum time the API waits for an import to complete.
orchestrator.api.autoscaling — Min/max replicas and CPU target for the orchestrator API.

Agent Sidecar

orchestrator.agent.tokenSecretName — Names the prerequisite opaque secret holding the bearer token that secures all internal communication between the orchestrator, the agents and the UI. The secret must exist before you install (see Prerequisites).
orchestrator.agent.import.batchSize — Number of rows per INSERT statement during imports (default: 20000).
orchestrator.agent.recovery — Retry settings for Galera cluster recovery after a split-brain event.

orchestrator.ui.publicAccess.enabled — Defaults to false. The UI has no authentication of its own and injects the admin token for any caller, so turning this on publishes full cluster admin to the internet.
orchestrator.ui.internalAccess.type — Inbound scope for in-GVC callers: same-gvc (default), same-org, workload-list or none. With workload-list, name the permitted workloads in orchestrator.ui.internalAccess.workloads.

Backup (Optional)

Set orchestrator.backup.enabled: true to enable scheduled backups to S3.

Scheduled backups did not work before 2.1.0. On every earlier version the backup job failed on every run and nothing ever reached the bucket, so an install with backups enabled had none. If you are upgrading, verify that objects appear in your bucket after the first scheduled run.

Restore is driven through the orchestrator API or UI — there is no orchestrator.action: restore.

orchestrator.backup.cloudAccountName — Cloud Account with write access to the backup S3 bucket.
orchestrator.backup.s3Bucket / s3Region — Backup bucket name and region.
orchestrator.backup.s3Policy — Custom IAM policy name(s) granting write access to the backup bucket (see Prerequisites).
orchestrator.backup.dataSet — The table dataset to back up.
orchestrator.backup.prefix — S3 folder prefix for backup archives.
orchestrator.backup.schedules — List of backup schedules. Each entry specifies a table, a backup type (delta for incremental, main for full), and a schedule in cron format.

Domain (Optional)

domain.enabled — Enable or disable the domain resource.
domain.name — Fully qualified domain name (e.g., manticore.example.com). Requires DNS configuration pointing to the GVC’s load balancer.
domain.dnsMode — DNS routing mode: cname for subdomain-based routing or ns for zone delegation.

When enabled, the domain routes /api/* to the orchestrator API and all other traffic to the UI.

Load Testing (Optional)

loadTest.enabled — Enable or disable the k6 load test workload.
loadTest.vus — Number of virtual users.
loadTest.duration — Total test duration (e.g., 30s, 5m, 1h).
loadTest.rps — Target requests per second (null = unlimited).
loadTest.target.endpoint — Search endpoint to target: search (JSON body) or sql.
loadTest.thresholds.p95ResponseTime — P95 response time threshold in milliseconds.
loadTest.thresholds.errorRate — Maximum acceptable error rate (e.g., 0.01 = 1%).

Upgrading From 2.0.1 or Earlier

2.0.1 and earlier shipped a live working bearer token as a values default, published in this repository, so every install that did not override it shared one publicly-known credential. The management UI was also public by default while having no login of its own, and the chart’s secret policy granted reveal on every secret in the organization rather than the four it needed. Treat the token on any earlier install as compromised: rotate it rather than simply upgrading.

2.0.1 and earlier	2.1.0
`orchestrator.agent.token` (a working value)	`orchestrator.agent.tokenSecretName` → prerequisite opaque secret
`orchestrator.ui.allowExternalAccess: true`	`orchestrator.ui.publicAccess.enabled: false`
UI internal scope hardcoded	`orchestrator.ui.internalAccess.type` (`same-gvc`)
Policy granted `reveal` org-wide	Scoped to exactly four secrets
Scheduled backup failed on every run	Backup works; restore via API/UI
Source-bucket IAM asked for write	Read-only is sufficient

Both removed keys stop the render with a message naming their replacement, so an upgrade cannot silently fall back to a default:

orchestrator.agent.token was REMOVED in manticore 2.1.0. The bearer token is now a
prerequisite opaque secret you create before install; set
orchestrator.agent.tokenSecretName to its name.

Create the agent-token secret

Generate a fresh token — do not carry the old one forward — and create the secret as shown in Prerequisites.

Replace the removed keys

Delete orchestrator.agent.token and orchestrator.ui.allowExternalAccess from your values, then set orchestrator.agent.tokenSecretName. If you were relying on external UI access, decide deliberately whether to set orchestrator.ui.publicAccess.enabled: true — it publishes cluster admin.

Narrow the source-bucket policy

If you granted s3:PutObject or s3:DeleteObject on your source data bucket, you can reduce it to read-only.

Expect a long rollout

A helm upgrade restarts the cluster serially and takes around nine minutes even when nothing has changed. Data survives it.

Operational Notes

A no-op helm upgrade takes roughly 542 seconds. The stateful tier restarts serially, so an upgrade that changes nothing still takes about nine minutes. This is expected, not a hang.
Verify per-replica row counts after a restore. A restore followed by a rolling restart has been observed to land on some replicas and not others while every node reported synced — so cluster status alone is not sufficient confirmation. Check counts on each replica.
Multi-segment imports and clusterMain: true are unverified at the shipped resource sizes. Testing at reduced CPU and memory saw Galera replication timeouts, but that is not a finding about the feature itself; it has not been exercised at the default 4 CPU / 8 Gi. Treat these as untested rather than known-good or known-broken.
k6 load-test thresholds now appear in cpln logs. Earlier versions exited before the summary was emitted, so results were unreadable.

External References