Manticore Search

​

Overview

​

What Gets Created

• Manticore Search Workload — A stateful Galera cluster with configurable replicas. Each replica runs a sidecar agent that handles cluster coordination, data imports, and recovery operations.
• Orchestrator API Workload — A REST API service for triggering imports, monitoring cluster health, initiating repairs, and managing backup and restore operations.
• Orchestrator Cron Job — A scheduled workload that runs import, health check, or repair operations on a configured schedule.
• UI Workload — A web dashboard for managing the cluster, monitoring replication, triggering operations, and visualizing queries.
• Volume Set — Persistent storage allocated per replica for Manticore data directories.
• Shared Volume Set — A shared volume accessible by all replicas and the orchestrator, used for slot-based import coordination.
• Secrets — Four opaque secrets: the Manticore searchd base configuration, the startup/shutdown handler script, the table schema registry used by the agent, and the agent bearer token. A K6 load test script secret is also created when load testing is enabled.
• Identities & Policies — Identities for the main cluster, orchestrator, and backup workloads. Policies grant reveal access to the configuration secrets, exec permissions on the orchestrator cron job, and view access to the Manticore workload for orchestration.
• Backup Cron Job (optional) — Scheduled logical backups of delta and full tables to an S3 bucket. Enabled when orchestrator.backup.enabled: true.
• Domain (optional) — Routes /api/* to the orchestrator API and all other traffic to the UI. Enabled when domain.enabled: true.
• Load Test Workload (optional) — A k6-based load test runner with a controller for automated scheduling. Enabled when loadTest.enabled: true.

​

Prerequisites

​

AWS S3 (Source Data)

1. Create an S3 bucket and upload your CSV source files. Set buckets.sourceBucket to the bucket name and buckets.awsRegion to its region.
2. If you do not have a Control Plane Cloud Account set up, follow the Create a Cloud Account guide. Set buckets.cloudAccountName to the name of your Cloud Account.
3. Set buckets.awsPolicyRefs to the IAM policies granting S3 access. For read-only source access, use the AWS managed policy aws::AmazonS3ReadOnlyAccess. For custom policies, omit the aws:: prefix.

​

Agent Token

openssl rand -base64 32

​

AWS S3 (Backup, Optional)

1. Create a separate S3 bucket for backups. Set orchestrator.backup.s3Bucket and orchestrator.backup.s3Region.
2. Create an IAM policy with the following permissions, replacing YOUR_BUCKET_NAME:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::YOUR_BUCKET_NAME",
                "arn:aws:s3:::YOUR_BUCKET_NAME/*"
            ]
        }
    ]
}

3. Set orchestrator.backup.cloudAccountName and orchestrator.backup.s3Policy to the name of your Cloud Account and the custom policy created above.

​

Installation

​

Configuration

# =============================================================================
# AWS Cloud Account and S3 Configuration
# =============================================================================
buckets:
  cloudAccountName: my-cloud-account    # Name of your configured Cloud Account
  awsPolicyRefs:                        # IAM policies for S3 access
    - aws::AmazonS3ReadOnlyAccess       # Note: if using a custom policy, omit the aws:: prefix
  awsRegion: us-east-1                  # Region of your S3 bucket
  sourceBucket: my-bucket               # S3 bucket containing files to import

# =============================================================================
# Tables Configuration
# =============================================================================
tables:
  - name: addresses
    csvPath:
      - manticore-imports/addresses/data.csv
    config:
      haStrategy: noerrors
      agentRetryCount: 3
      clusterMain: false
      segmentCount: 1
      charsetTable: non_cont
      memLimit: 2G
      hasHeader: true
    schema:
      columns:
        - name: address_id
          type: attr_uint
        - name: street_number
          type: attr_uint
        - name: street_name
          type: field
        - name: city
          type: field
        - name: county
          type: field
        - name: state
          type: field
        - name: postal_code
          type: field
        - name: country
          type: field
        - name: latitude
          type: attr_float
        - name: longitude
          type: attr_float

# =============================================================================
# Manticore Search Configuration
# =============================================================================
manticore:
  image: manticoresearch/manticore:15.1.0
  clusterName: manticore          # Galera cluster name
  resources:
    cpu: 4
    memory: 8Gi
  volumeset:
    capacity: 200                 # GB per replica
  sharedVolumeset:
    capacity: 100                 # GB shared across replicas and orchestrator

  # minScale = replica count used by orchestrator for coordination
  autoscaling:
    minScale: 3
    maxScale: 4
    metric: rps                   # rps for stateful, cpu for standard
    target: 100
    scaleToZeroDelay: 300

  rolloutOptions:
    maxSurgeReplicas: 25%
    maxUnavailableReplicas: '0'
    minReadySeconds: 10
    scalingPolicy: OrderedReady
    terminationGracePeriodSeconds: 60

  # IMPORTANT: same-gvc required for Galera replication
  firewall:
    internalAccess:
      type: same-gvc
      workloads: []

# =============================================================================
# Orchestrator Configuration
# =============================================================================
orchestrator:
  version: v5.1.0
  image: ghcr.io/controlplane-com/manticore-orchestrator/manticore-cpln-api
  logLevel: debug                 # debug, info, warn, error
  resources:
    cpu: 1
    memory: 2Gi

  # Cron job settings
  schedule: "0 * * * *"           # Cron schedule (default = every hour)
  action: import                  # init, import, health, repair
  tableName: addresses            # Must match a name in tables[]
  suspend: true                   # Start suspended (trigger via UI/API)
  timeoutSeconds: 900             # Container timeout (default 15 minutes)
  importMemLimit: 2G              # Memory limit for import jobs
  activeDeadlineSeconds: 14400    # Max job runtime (default 4 hours)

  # Orchestrator API
  api:
    version: v5.1.0
    image: ghcr.io/controlplane-com/manticore-orchestrator/manticore-cpln-api
    logLevel: debug
    importPollInterval: 30s
    importPollTimeout: 2h
    resources:
      cpu: 0.25
      memory: 256Mi
    autoscaling:
      maxScale: 3
      minScale: 2
      metric: cpu
      target: 80

  # Agent sidecar
  agent:
    version: v5.1.0
    image: ghcr.io/controlplane-com/manticore-orchestrator/manticore-cpln-agent
    # REQUIRED: Generate with `openssl rand -base64 32`
    token: "6Gl5uO9KkKAh1u+ymoBW98WCtjTFpljuhpLdKb+tNAA="
    resources:
      cpu: 250m
      minCpu: 100m
      memory: 512Mi
      minMemory: 128Mi
    import:
      batchSize: 20000            # Rows per INSERT statement
    recovery:
      maxRetries: 5               # Retry attempts for cluster recovery
      initialBackoffSec: 5        # Initial delay between retries
      maxBackoffSec: 60           # Max backoff delay (exponential)

  # Web UI
  ui:
    version: v5.1.0
    image: ghcr.io/controlplane-com/manticore-orchestrator/manticore-cpln-ui
    resources:
      cpu: 0.25
      memory: 0.25Gi
    allowExternalAccess: true     # false = internal GVC access only
    autoscaling:
      maxScale: 2
      minScale: 1
      metric: cpu
      target: 80

  # Backup Configuration (Optional)
  backup:
    enabled: false
    version: v5.1.0
    image: ghcr.io/controlplane-com/manticore-orchestrator/manticore-cpln-backup
    cloudAccountName: my-backup-cloud-account
    s3Bucket: my-backup-bucket
    s3Policy:
      - my-backup-policy
    s3Region: us-east-1
    dataSet: addresses            # Data set to back up
    prefix: manticore-backups     # S3 prefix/folder for backups
    schedules:
      - {"table":"addresses","type":"delta","schedule":"0 2 * * *"}   # Daily at 2am UTC
      - {"table":"addresses","type":"main","schedule":"0 2 1 * *"}    # Monthly on 1st at 2am UTC
    activeDeadlineSeconds: 14400
    resources:
      cpu: 1
      memory: 1Gi

# =============================================================================
# Domain Configuration (Optional)
# =============================================================================
domain:
  enabled: false
  name: ""                        # FQDN, e.g., manticore.example.com
  dnsMode: cname                  # cname (subdomains) or ns (zone delegation)

# =============================================================================
# Load Testing Configuration (Optional)
# =============================================================================
loadTest:
  enabled: false
  image: grafana/k6:0.47.0
  resources:
    cpu: 0.5
    memory: 512Mi
  vus: 10                         # Virtual users
  duration: "5m"                  # Test duration (e.g., 30s, 5m, 1h)
  rps: null                       # Target RPS (null = unlimited)
  replicas: 1
  controller:
    image: alpine/curl
    schedule: ""                  # Cron expression (empty = manual only)
    testDurationBuffer: 60        # Seconds added to duration before scale-down
  target:
    port: 9308                    # Manticore HTTP API port
    endpoint: search              # "search" or "sql"
  query:
    index: addresses
    query:
      match:
        "*": "test"
    limit: 10
  thresholds:
    p95ResponseTime: 500          # ms
    errorRate: 0.01               # 1%

​

S3 and Cloud Account

• buckets.cloudAccountName — Name of the Control Plane Cloud Account with AWS trust configured.
• buckets.awsPolicyRefs — IAM policies granting read access to the source S3 bucket. Use aws::AmazonS3ReadOnlyAccess for the AWS managed policy, or omit the aws:: prefix for custom policies.
• buckets.awsRegion — AWS region of the source S3 bucket.
• buckets.sourceBucket — Name of the S3 bucket containing CSV files to import.

​

Tables

• name — Table name used by Manticore and referenced by the orchestrator.
• csvPath — List of S3 paths (relative to sourceBucket) for the CSV source files. Multiple paths create a distributed multi-segment table.
• config.haStrategy — High-availability behavior when agents are unreachable (noerrors ignores unreachable agents).
• config.agentRetryCount — Number of retry attempts when an agent doesn’t respond.
• config.segmentCount — Number of distributed segments for this table. Use more than 1 for very large datasets split across multiple CSVs.
• config.memLimit — Memory limit for the Manticore index (e.g., 2G).
• config.hasHeader — Set to true if the CSV file includes a header row.
• schema.columns — Column definitions. Each column has a name and a type.

​

Manticore Cluster

• manticore.clusterName — Galera cluster name used for replication coordination.
• manticore.resources — CPU and memory for each Manticore replica.
• manticore.volumeset.capacity — Persistent storage per replica in GB.
• manticore.sharedVolumeset.capacity — Shared storage in GB, accessible by all replicas and the orchestrator for import slot coordination.
• manticore.autoscaling.minScale — Minimum replica count. This value is also used by the orchestrator to determine cluster quorum. It should match your intended fixed replica count.
• manticore.autoscaling.maxScale — Maximum replica count for autoscaling.

​

Orchestrator

• orchestrator.schedule — Cron schedule for automated operations (default: every hour).
• orchestrator.action — Operation to run: init (initial cluster setup), import (load CSV data), health (check cluster state), or repair (fix split-brain issues).
• orchestrator.tableName — The table to target for the cron job. Must match a name in tables.
• orchestrator.suspend — When true, the cron job is created in a suspended state and must be triggered manually via the UI or CLI. Recommended for production.
• orchestrator.timeoutSeconds — Timeout for each cron container execution.
• orchestrator.activeDeadlineSeconds — Maximum total runtime for an import job before it is terminated.

• orchestrator.api.importPollInterval — How often the API checks import progress.
• orchestrator.api.importPollTimeout — Maximum time the API waits for an import to complete.
• orchestrator.api.autoscaling — Min/max replicas and CPU target for the orchestrator API.

• orchestrator.agent.token — Bearer token securing all internal communication between the orchestrator and agent sidecars. Generate with openssl rand -base64 32 and change before deploying.
• orchestrator.agent.import.batchSize — Number of rows per INSERT statement during imports (default: 20000).
• orchestrator.agent.recovery — Retry settings for Galera cluster recovery after a split-brain event.

• orchestrator.ui.allowExternalAccess — When true, the UI is accessible from the internet. Set to false to restrict access to within the GVC.

​

Backup (Optional)

• orchestrator.backup.cloudAccountName — Cloud Account with write access to the backup S3 bucket.
• orchestrator.backup.s3Bucket / s3Region — Backup bucket name and region.
• orchestrator.backup.s3Policy — Custom IAM policy name(s) granting write access to the backup bucket (see Prerequisites).
• orchestrator.backup.dataSet — The table dataset to back up.
• orchestrator.backup.prefix — S3 folder prefix for backup archives.
• orchestrator.backup.schedules — List of backup schedules. Each entry specifies a table, a backup type (delta for incremental, main for full), and a schedule in cron format.

​

Domain (Optional)

• domain.enabled — Enable or disable the domain resource.
• domain.name — Fully qualified domain name (e.g., manticore.example.com). Requires DNS configuration pointing to the GVC’s load balancer.
• domain.dnsMode — DNS routing mode: cname for subdomain-based routing or ns for zone delegation.

​

Load Testing (Optional)

• loadTest.enabled — Enable or disable the k6 load test workload.
• loadTest.vus — Number of virtual users.
• loadTest.duration — Total test duration (e.g., 30s, 5m, 1h).
• loadTest.rps — Target requests per second (null = unlimited).
• loadTest.target.endpoint — Search endpoint to target: search (JSON body) or sql.
• loadTest.thresholds.p95ResponseTime — P95 response time threshold in milliseconds.
• loadTest.thresholds.errorRate — Maximum acceptable error rate (e.g., 0.01 = 1%).

​

External References