Skip to main content

Overview

Manticore Search is an open-source full-text search engine. This template deploys a replicated Manticore cluster using Galera replication, together with an orchestrator that performs coordinated, zero-downtime CSV imports from an S3 bucket into every replica, optional scheduled backups to S3, and a web dashboard for cluster operations. It is built for datasets that are regenerated upstream and republished as CSV: data arrives in a bucket, and the orchestrator turns it into a distributed table that keeps serving searches while it is reloaded.
This template deploys into an existing GVC that you already have. It does not create or manage a GVC.

What Gets Created

Prerequisites

Two things must exist before you install: the agent-token secret and the S3 source bucket with a cloud account that can read it. Only AWS S3 is supported.
1

Create the agent-token secret

The orchestrator, the agent sidecars and the UI authenticate to each other with one bearer token, stored in an opaque secret you create:
Set orchestrator.agent.tokenSecretName to this name.
If the secret does not exist at install time, the deployment wedges silently: the containers never start, cpln logs returns nothing, and the only place the missing secret is named is status.versions[].message:
Create the secret and the deployment recovers on its own; to skip the wait, run cpln workload force-redeployment RELEASE_NAME-manticore --gvc GVC_NAME.
2

Create the source bucket and a read-only IAM policy

Create an S3 bucket and upload your CSV files (for example to imports/addresses.csv). Imports only ever read from this bucket — every scratch artifact is written to the shared volume — so a read-only policy is sufficient. Create an IAM policy with the following JSON, replacing my-manticore-bucket:
Set buckets.sourceBucket to the bucket name and buckets.awsRegion to its region.
3

Create a cloud account and reference the policy

Create a Control Plane cloud account for your AWS account by following the Create a Cloud Account guide, and attach the policy above to the role it uses. Set buckets.cloudAccountName to the cloud account name and buckets.awsPolicyRefs to the policy name. Custom policies are named without a prefix; only AWS-managed policies take the aws:: prefix (for example aws::AmazonS3ReadOnlyAccess).
Scheduled backups need a second, writable bucket with its own policy and cloud account. That setup is optional and described under Backing Up.

Installation

Install from the marketplace registry into your existing GVC, naming the agent-token secret and the source-bucket resources you created above:
The default tables value defines one example table, addresses, read from imports/addresses.csv. Define your own tables in a values file (see Tables) and pass it with -f as described in the CLI installation guide.

UI

Browse, install, and manage templates visually

CLI

Manage templates from your terminal

Terraform

Declare templates in your Terraform configurations

Pulumi

Declare templates in your Pulumi programs

Configuration

Buckets

Tables

One entry per searchable table. csvPath is a path within buckets.sourceBucket, or a list of paths for a multi-segment table.
If the first CSV column is numeric it becomes the document ID and must not be declared in schema.columns; otherwise an ID is generated. Multi-segment tables. To split a large dataset across several CSV files and query them as one distributed table, set csvPath to a list and segmentCount to its length. The chart refuses to render if they disagree. A backup of a multi-segment table covers all segments as one archive, and a restore replaces all of them.
Multi-segment imports and clusterMain: true have not been verified at the shipped resource sizes. Treat them as untested rather than known-good, and validate per-replica row counts after an import if you rely on either.

Manticore

Orchestrator

The orchestrator block configures the cron job, the REST API, the agent sidecar, the web UI and the optional backup job.
The management UI has no authentication of its own. It holds the bearer token server-side and attaches it to every request it makes on a visitor’s behalf, so anyone who can reach the UI holds full cluster admin — imports, restores and repairs included. The token protects the orchestrator API, not the UI. Keep orchestrator.ui.publicAccess.enabled: false and reach the UI from inside the GVC, or place your own authenticating proxy in front of it before exposing it by any route, including a custom domain.

Domain

When enabled, the domain routes /api/* to the orchestrator API (bearer token required) and everything else to the UI (no authentication — see the warning above).

Load Test

How to run a test and read its results is described under Load Testing.

Connecting

The bearer token is the value you stored in the secret named by orchestrator.agent.tokenSecretName:
Verify the cluster from inside a Manticore replica. The manticore container ships the mysql client, and searchd listens on loopback, so this works from any replica (substitute your manticore.clusterName for manticore in the status key if you changed it):
A healthy cluster reports primary. To use a MySQL client on your own machine instead, forward the SQL port and connect to 127.0.0.1:9306:

First Run

An install indexes nothing: the tables you defined exist but are empty until the cluster has been initialized once and an import has run. Both are orchestrator actions. Run them from the UI (reachable from inside the GVC on port 3000), or from the CLI by pointing the cron job at the action and starting it — see Running Cluster Actions. Run init once, then import; repeat import whenever the CSVs in your source bucket change.

Operations

Running Cluster Actions

The RELEASE_NAME-orchestrator-job cron workload ships suspended. orchestrator.action selects what a run does — init bootstraps the Galera cluster, import loads the CSVs for orchestrator.tableName into a fresh slot and swaps it in without interrupting searches, health reports cluster state, and repair recovers from a split brain. Trigger a run on demand:
The UI exposes the same actions plus backup and restore. Follow a run in the job’s logs:
To run on a schedule instead, set orchestrator.suspend: false; orchestrator.schedule then applies.

Backing Up

Backups are logical delta (incremental) and main (full) archives written to a separate S3 bucket by the RELEASE_NAME-manticore-backup cron workload, under its own identity. Do not widen the source-bucket policy; give backups their own bucket and policy.
1

Create the backup bucket and a writable policy

Create a second bucket and an IAM policy with the following JSON, replacing my-manticore-backup-bucket. GetObject and DeleteObject let restores read an archive back and lifecycle cleanup remove one; a backup-only install can drop them.
2

Attach it to a cloud account and enable backups

Attach the policy to the role your cloud account uses, then set orchestrator.backup.enabled: true together with orchestrator.backup.cloudAccountName, s3Bucket, s3Region and s3Policy.
orchestrator.backup.schedules drives the automated runs (daily delta and monthly main by default). You can also run a backup on demand from the UI, or from any workload inside the GVC against the orchestrator API:
After enabling backups, confirm that objects appear in the bucket after the first scheduled run.

Restoring a Backup

Restore is driven through the UI or the orchestrator API; there is no orchestrator.action: restore. List the available archives (type defaults to delta, so pass type=main to see full backups), then restore one by filename:
A restore scales the cluster up by one replica while it runs and puts it back afterwards. After restoring a main backup, use the UI’s Rotate Main control to swap the active slot.
A restore can land on some replicas and not others while every node still reports synced, so cluster status alone does not confirm it. Count rows on each replica (the stateful replicas are named RELEASE_NAME-manticore-0, -1, -2, …):
If the counts disagree, run a fresh import of the table from your source CSVs, which rebuilds every replica.

Upgrading from 2.0.1 or Earlier

2.0.1 and earlier shipped a live working bearer token as a values default, published in the template repository, so every install that did not override it shared one publicly-known credential. The management UI was also public by default while having no login of its own, and the chart’s secret policy granted reveal on every secret in the organization rather than the four it needed. Treat the token on any earlier install as compromised: rotate it rather than simply upgrading. Both removed keys stop the render with a message naming their replacement, so an upgrade cannot silently fall back to a default:
1

Create the agent-token secret

Generate a fresh token — do not carry the old one forward — and create the secret as shown in Prerequisites.
2

Replace the removed keys

Delete orchestrator.agent.token and orchestrator.ui.allowExternalAccess from your values, then set orchestrator.agent.tokenSecretName. If you were relying on external UI access, decide deliberately whether to set orchestrator.ui.publicAccess.enabled: true — it publishes cluster admin.
3

Narrow the source-bucket policy

If you granted s3:PutObject or s3:DeleteObject on your source data bucket, reduce it to the read-only policy shown in Prerequisites.
4

Verify backups if you use them

Scheduled backups did not work on any earlier version, so an install with backups enabled had none. After upgrading, confirm that objects appear in the backup bucket after the first scheduled run.

Upgrading from 2.1.0

The agent sidecar’s resource block exposes both a reservation and a limit, so 2.2.0 names the limit maxCpu / maxMemory instead of cpu / memory. Rename the two keys under orchestrator.agent.resources in your values: minCpu and minMemory are unchanged, and blocks that expose only a limit (manticore.resources, orchestrator.resources, orchestrator.api.resources, orchestrator.ui.resources, orchestrator.backup.resources, loadTest.resources) keep the bare cpu / memory names. An upgrade that still carries the old names is refused at render:

Scaling and Availability

  • manticore.autoscaling.minScale is both the minimum replica count and the replica count the orchestrator coordinates imports across. Changing it after the cluster is initialized requires a repair run.
  • Every helm upgrade — including one that changes nothing — restarts the Manticore replicas one at a time (scalingPolicy: OrderedReady). Indexed data survives, and searches keep serving from the replicas that are still up, but budget for the full serial rollout. Editing tables changes a mounted secret and therefore triggers the same rollout.
  • manticore.firewall.internalAccess.type must stay same-gvc: Galera replication between replicas depends on it, and Manticore’s SQL and HTTP ports have no authentication of their own, so they rely on that firewall. Allow a few minutes for any access change to take effect.

Load Testing

With loadTest.enabled: true, trigger a run by starting the controller; it scales the k6 workload up to loadTest.replicas, waits duration plus testDurationBuffer, then scales it back to zero:
Read the results — including whether each threshold passed — from the runner’s logs:
The last lines are k6’s end-of-test summary (http_req_duration, http_req_failed, a pass/fail mark per threshold) followed by k6 exited with status N — non-zero means a threshold was breached. Set loadTest.controller.schedule to a cron expression to run tests automatically.

Troubleshooting

Cause: the agent-token secret named by orchestrator.agent.tokenSecretName did not exist at install time. The containers never start, so there is no log output; the only place the missing secret is named is status.versions[].message.Fix: read the message, create the secret, and the deployment recovers on its own (or force a redeployment to skip the wait):
Cause: your values still carry a key from 2.0.1 or earlier. The same guard reports orchestrator.ui.allowExternalAccess was REMOVED in manticore 2.1.0.Fix: follow Upgrading from 2.0.1 or Earlier.
Cause: your values still use the 2.1.0 names for the agent sidecar’s limits. The same guard reports orchestrator.agent.resources.memory was RENAMED to orchestrator.agent.resources.maxMemory.Fix: rename the two keys as shown in Upgrading from 2.1.0.
Cause: a table’s config.segmentCount does not equal the number of csvPath entries (or csvPath is a single string while segmentCount is not 1).Fix: set segmentCount to the number of CSV paths for that table.
Cause: an install alone indexes nothing. The tables exist but are empty until the cluster has been initialized and an import has run.Fix: run init once and then import as described in Running Cluster Actions, then confirm with SELECT COUNT(*) on the table.
Cause: a restore can land on some replicas and not others while every node still reports synced.Fix: count rows on each replica as shown in Restoring a Backup; if they disagree, run a fresh import of the table, which rebuilds every replica from the source CSVs.

Important Notes

  • Create the agent-token secret before installing; without it the deployment wedges silently and status.versions[].message is the only diagnostic.
  • Enabling orchestrator.ui.publicAccess or routing a domain to the UI puts an unauthenticated admin console on the public internet. Only do it behind your own authenticating proxy.
  • Keep manticore.firewall.internalAccess.type at same-gvc; Galera replication requires it and Manticore’s 9306/9308 ports have no authentication.
  • Tables are empty until you run init followed by import; the install alone does not load data.
  • The source bucket only ever needs read access. Only the optional backup bucket needs write, and it has its own identity and policy.
  • Every helm upgrade restarts all Manticore replicas one at a time; data survives, but plan the rollout window.
  • Verify per-replica row counts after a restore rather than trusting cluster status.
  • Uninstalling deletes both volume sets and all indexed data. Take a backup first if you need it.

External References

Manticore Search Documentation

Official Manticore Search manual

Replication Setup

How Manticore’s Galera-based replication works

Manticore Search GitHub

Manticore Search open-source repository

Orchestrator Source

Orchestrator, agent, UI and backup images used by this template

Manticore Template

View the source files, default values, and chart definition