DuckDB

Overview

DuckDB is an in-process analytical SQL engine that reads and writes Parquet, CSV and JSON directly, queries object storage over the S3 API, and can attach live PostgreSQL, MySQL and SQLite databases. This template runs a SQL script of yours on a cron schedule and exits: it is a batch job runner, billed only for the minutes a run is executing, with nothing running between runs.

This is a scheduled job runner, not a query service. The workload is a cron job: it starts, runs your script, and terminates. It binds no port, has no endpoint, and nothing is listening after install — that is correct behavior, not a broken deployment. If you want an always-on SQL endpoint that BI tools and JDBC clients connect to, install trino instead.

This template deploys into an existing GVC that you already have; it does not create or manage one. A cron workload runs in every location the GVC has, so a GVC with several locations runs this job once per location on each schedule.

What Gets Created

Resource	Name	Purpose	Optional
Workload	`RELEASE_NAME-duckdb`	Cron job running `duckdb/duckdb:1.5.5` once per schedule; no ports, no inbound traffic	—
Secret	`RELEASE_NAME-duckdb-preamble`	Generated `SET` statements (memory, threads, scratch paths, object-store secret) run before your script	—
Secret	`RELEASE_NAME-duckdb-script`	Your SQL from `sql.inline`, mounted at `/etc/duckdb/job.sql`	Not created when `sql.secretName` is set
Identity	`RELEASE_NAME-duckdb-identity`	Workload identity; carries the AWS cloud-account binding when `objectStore.type: aws`	—
Policy	`RELEASE_NAME-duckdb-policy`	Grants the identity `reveal` on exactly the secrets this release mounts	—

No volume set is created. Each run starts from an empty in-memory database, and extensions and query spill live in container-local scratch under /tmp.

Prerequisites

None for a default install. The shipped sql.inline is a self-test that needs no credentials, no bucket and no cloud account. Every run does download its extensions from extensions.duckdb.org:443; the template allows all outbound traffic, so this works unless your organization restricts egress. A real job has to write its results somewhere durable — object storage or an attached database — and the features that make that possible each need something created before install:

Object storage (when objectStore.type is aws or s3-compatible)

Anything other than none registers a DuckDB S3 secret so your SQL can read and write s3:// paths.

AWS S3 (keyless)
S3-compatible

The job receives short-lived credentials from its own workload identity — no access keys are created, stored or injected anywhere.

Create your S3 bucket and note its region.
If you do not have one, create a cloud account for your AWS account.
Create an IAM policy scoped to that bucket (replace YOUR_BUCKET_NAME). Drop s3:PutObject and s3:DeleteObject if the job only reads.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation"
      ],
      "Resource": "arn:aws:s3:::YOUR_BUCKET_NAME"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Resource": "arn:aws:s3:::YOUR_BUCKET_NAME/*"
    }
  ]
}

Set objectStore.type: aws, objectStore.aws.region to the bucket’s region, objectStore.aws.cloudAccountName to the cloud account’s name, and objectStore.aws.policyName to the IAM policy’s name. The chart refuses to render if any of the three is blank.

Covers SeaweedFS, MinIO, Cloudflare R2, Tigris, and Google Cloud Storage through its S3 interoperability endpoint. These servers cannot federate with a cloud account, so they use static keys held in a dictionary secret.

Create the bucket on your server, plus an access key and secret key pair with read and write access to it. For Google Cloud Storage these are HMAC keys, created under Cloud Storage → Settings → Interoperability.
Create the credentials secret with exactly these two keys:

cpln secret create-dictionary --name SECRET_NAME \
  --entry access-key-id=YOUR_ACCESS_KEY \
  --entry secret-access-key=YOUR_SECRET_KEY

Set objectStore.type: s3-compatible, objectStore.s3Compatible.credentialsSecretName to the name you used, and objectStore.s3Compatible.endpoint to the server’s host:port with no http:// or https:// prefix — the chart rejects a scheme and tells you to set useSsl instead.

Provider	`endpoint`	`urlStyle`	`useSsl`
SeaweedFS in the same GVC	`RELEASE_NAME-seaweedfs.GVC_NAME.cpln.local:8333`	`path`	`false`
MinIO in the same GVC	`RELEASE_NAME-minio.GVC_NAME.cpln.local:9000`	`path`	`false`
Google Cloud Storage	`storage.googleapis.com`	`path`	`true`
Cloudflare R2	`ACCOUNT_ID.r2.cloudflarestorage.com`	`path`	`true`
AWS S3 with static keys	`s3.us-east-1.amazonaws.com`	`vhost`	`true`

The RELEASE_NAME in the SeaweedFS and MinIO rows is the release name of that template’s install, not this one.

A script secret (when using sql.secretName instead of sql.inline)

Put your SQL in an opaque secret with --encoding plain; its payload is mounted as the job script:

cat job.sql | cpln secret create-opaque --name SECRET_NAME --encoding plain -f -

Set sql.secretName to this name. The template’s own script secret is then not created at all.

Credential secrets for attached databases (when using secretEnv)

Every secret named in secretEnv[].secretName must exist before install. For a postgres or mysql template in the same GVC, that is the credentials secret you created for it — point secretKey at its password entry.

Create every referenced secret before installing, or the job wedges silently. A sql.secretName, secretEnv[].secretName or credentialsSecretName pointing at a secret that does not exist installs “successfully”, and the container then never starts — so cpln logs returns zero lines. The reason is named in status.versions[].message:

cpln workload get-deployments RELEASE_NAME-duckdb --gvc GVC_NAME -o yaml

Use get-deployments — plain cpln workload get has no versions key. Once the secret exists the workload recovers on its own after several minutes, or run cpln workload force-redeployment RELEASE_NAME-duckdb --gvc GVC_NAME to skip the wait.

Installation

A default install needs no --set values — it runs the built-in self-test script on the default schedule:

cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/duckdb \
  --version 1.0.0 --gvc GVC_NAME

Or follow the instructions for your preferred method:

UI

Browse, install, and manage templates visually

CLI

Manage templates from your terminal

Terraform

Declare templates in your Terraform configurations

Pulumi

Declare templates in your Pulumi programs

Configuration

Image and Schedule

image: duckdb/duckdb:1.5.5

schedule: "0 2 * * *" # cron expression in UTC — always quote it (e.g. "*/15 * * * *")
suspend: false # true = never run automatically; start it with `cpln workload cron start`
activeDeadlineSeconds: 3600 # a run still going is terminated within ~2 min of this

The schedule must be exactly five fields; the chart rejects anything else at render time. Runs never overlap and a failed run is not retried — the next scheduled run simply starts as normal.

activeDeadlineSeconds is not a hard cut. The deadline is detected on time, but the container is terminated within roughly two minutes of it and keeps consuming its full CPU and memory allocation for that window. A run killed this way can still print the success marker on its way out — see Confirming a Run Succeeded.

SQL Script

sql:
  inline: |
    -- Default self-test. Replace with your own transform.
    SELECT 'duckdb-template-ok' AS status, version() AS duckdb_version, now() AS run_at;
  secretName: "" # opaque secret (encoding plain, payload = SQL) used INSTEAD of inline, e.g. my-duckdb-script

sql.secretName wins when both are set, and is the right choice when the script is too large for a values file or you want to change it without a Helm upgrade. The chart refuses to render when both are empty. Before your script, the job runs a template-generated preamble containing .bail on and the derived memory_limit, threads, temp_directory and extension_directory settings, then the object-store secret when one is configured. Because the preamble runs first, a SET in your own script always wins. .bail on stops execution at the first SQL error.

Credentials in SQL

secretEnv: []
# secretEnv:
#   - name: PGPASSWORD
#     secretName: my-postgres-credentials # must exist BEFORE install
#     secretKey: password # omit for an opaque secret (uses its payload)

Each entry becomes a container environment variable sourced from a Control Plane secret, readable in your SQL as getenv('NAME'). Only a cpln://secret/... reference appears in the rendered workload; the value itself never lands in your values file. Names must be UPPER_SNAKE_CASE and unique, and AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN and AWS_DEFAULT_REGION are rejected because objectStore owns them. To attach a database, name the entry after the driver’s own password variable and leave the password out of the connection string — the driver reads it from the environment. ATTACH takes a string literal, so getenv() cannot be concatenated into it (see Troubleshooting).

Attach type	Name the `secretEnv` entry
`postgres`	`PGPASSWORD`
`mysql`	`MYSQL_PWD`

ATTACH 'dbname=postgres user=postgres host=RELEASE_NAME-postgres.GVC_NAME.cpln.local' AS pg (TYPE postgres, READ_ONLY);
COPY (SELECT * FROM pg.public.events) TO 's3://YOUR_BUCKET_NAME/events.parquet' (FORMAT parquet);

Here RELEASE_NAME-postgres is the workload of a postgres template installed in the same GVC. getenv('NAME') still works anywhere an ordinary expression is allowed — a WHERE clause, a computed column, a COPY destination built with ||.

Resources and Tuning

resources:
  minCpu: 500m
  maxCpu: 2000m # also sets DuckDB threads: one per whole core, minimum 1
  minMemory: 1Gi
  maxMemory: 4Gi

tuning:
  memoryLimitPercent: 60 # DuckDB memory_limit = this percent of maxMemory (20–80)

DuckDB reads the host machine’s RAM and core count rather than the container’s limits, so left alone it would size itself for hardware it does not have and get OOM-killed. The template derives both settings from your values and writes them into the preamble:

Setting	Derived from	At the defaults
`memory_limit`	`tuning.memoryLimitPercent` of `resources.maxMemory`	`2457MiB`
`threads`	One per whole core of `resources.maxCpu`, minimum 1	`2`

The chart validates this block at render time: minMemory may not exceed maxMemory, minCpu may not exceed maxCpu, maxCpu:minCpu may not exceed 4:1, and memoryLimitPercent must be between 20 and 80. Every job must fit in memory. No volume is attached, so size the work with resources.maxMemory and tuning.memoryLimitPercent rather than relying on spill. DuckDB’s out-of-core operators still function, but they spill to container-local scratch at /tmp/duckdb-temp, bounded by container disk, not to a sized, persistent volume. Extensions are written to /tmp/duckdb-extensions, which is likewise container-local, so they are downloaded again on every run; httpfs and aws autoload on first use of an s3:// path, so most scripts never issue an explicit INSTALL.

Object Storage

objectStore:
  type: none # options: none, aws, s3-compatible

  aws: # keyless — credentials come from the workload identity, no keys anywhere
    region: us-east-1
    cloudAccountName: my-s3-cloud-account # must exist BEFORE install
    policyName: my-duckdb-bucket-policy # IAM policy scoped to your bucket (see README)

  s3Compatible: # SeaweedFS, MinIO, Cloudflare R2, GCS interoperability, Tigris
    endpoint: my-seaweedfs.my-gvc.cpln.local:8333 # host:port, no http:// prefix
    region: us-east-1
    urlStyle: path # options: path, vhost
    useSsl: false
    credentialsSecretName: my-duckdb-s3-credentials # dictionary secret: access-key-id, secret-access-key

objectStore.type decides whether the preamble registers a DuckDB S3 secret; anything other than none lets your SQL read and write s3:// paths directly. The bucket, cloud account, IAM policy and credentials secret each mode needs are set up under Prerequisites. With s3-compatible, the identity is granted reveal on exactly the credentials secret you name, and the two keys are injected as AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

Connecting

This template exposes nothing to connect to — it is a job, not a server. Observe and drive it instead:

What	How
Public URL	None. The workload binds no port and accepts no inbound traffic.
Internal host:port	None.
Job output	`cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-duckdb"}' --limit 50 --since 10m`
Run history	`cpln workload cron get RELEASE_NAME-duckdb --gvc GVC_NAME`
Run it now	`cpln workload cron start RELEASE_NAME-duckdb --gvc GVC_NAME`
Credentials	None are issued. The job reads the secrets you name in `secretEnv` and `objectStore`.

To verify a fresh install, start a run by hand and look for the completion marker in its output:

cpln workload cron start RELEASE_NAME-duckdb --gvc GVC_NAME
cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-duckdb"} |= "duckdb-job-complete"' --limit 50 --since 10m

Log lines can take a few minutes to appear after a run finishes.

Operations

Confirming a Run Succeeded

A run succeeded only when both of these are true:

The line duckdb-job-complete appears in the job’s log output.
The run’s status is Successful in cpln workload cron get RELEASE_NAME-duckdb --gvc GVC_NAME.

Check both, never either one alone. Each covers a hole in the other:

The marker alone is not enough. A run terminated for exceeding activeDeadlineSeconds can finish its script inside the termination lag and print the marker on its way out, while the platform has already recorded the run as failed.
The status alone is not enough. DuckDB’s CLI can exit 0 on some failed scripts (upstream issue #16574), which would leave the run looking successful.

The marker is emitted by a final statement the template appends after your script, and the preamble’s .bail on stops execution at the first error — so a SQL error aborts the script and the marker is never printed.

Scaling and Availability

DuckDB is one process over one database, with no clustering and no failover in any edition, so there is deliberately no replicas knob: the workload is pinned to one replica per GVC location. Installing the template several times — different scripts, schedules or GVCs — is real scale-out, but it is not high availability. If a run’s container dies, that run did not happen, and a failed run is not retried; the next scheduled run starts normally. Nothing is stateful, so uninstalling and reinstalling is free: cpln helm uninstall RELEASE_NAME --gvc GVC_NAME removes everything the template created and leaves your prerequisite secrets in place.

Troubleshooting

Nothing is listening after install

Symptom: the install succeeds, but there is no endpoint, no port and nothing to connect to.Cause: this is a cron workload. It runs your script on the schedule and exits; nothing is meant to be listening.Fix: nothing to fix. Read run history with cpln workload cron get RELEASE_NAME-duckdb --gvc GVC_NAME and output with cpln logs. If you need an always-on SQL endpoint, install trino.

Parser Error: syntax error at or near "||"

Symptom: an ATTACH statement that builds its connection string with || getenv('PGPASSWORD') fails:

Parser Error: syntax error at or near "||"

Cause: ATTACH takes a string literal, not an expression. This applies equally to the postgres, mysql and sqlite attach types.Fix: name the secretEnv entry after the driver’s own password variable (PGPASSWORD for postgres, MYSQL_PWD for mysql) and leave password= out of the connection string. If a credential truly must sit inline, put the whole connection string into a sql.secretName secret instead.

The run was killed with no SQL error, or the container reports OOMKilled

Symptom: the run stops mid-script with no error in the logs, and cpln workload get-deployments RELEASE_NAME-duckdb --gvc GVC_NAME -o yaml shows an OOMKilled restart reason.Cause: the job needed more memory than the container has. The most common trigger is a SET memory_limit in your own script that exceeds what resources.maxMemory can back — your SET overrides the template’s derived limit.Fix: remove the SET memory_limit from your script and change tuning.memoryLimitPercent instead, or raise resources.maxMemory. Do not plan around spill; every job must fit in memory.

A run that worked yesterday now fails installing an extension

Symptom: a run fails at its first s3:// read or at an explicit INSTALL, with an HTTP or download error naming extensions.duckdb.org.Cause: there is no cache volume, so every run downloads its extensions from extensions.duckdb.org:443. The template allows all outbound traffic, so this usually means egress to that host was blocked at the organization or network level after the job was set up.Fix: restore outbound access to extensions.duckdb.org:443. The dependency is per run and cannot be removed by this template.

The run status is failed but the log shows duckdb-job-complete

Symptom: cpln workload cron get records the run as failed while the completion marker is in the output.Cause: the run exceeded activeDeadlineSeconds. The platform records the failure when the deadline passes, but terminates the container up to roughly two minutes later, and the script finished inside that lag.Fix: raise activeDeadlineSeconds if the run time is legitimate, or reduce the work. Treat this run as failed when deciding whether its output is complete.

helm install is refused with a message starting duckdb:

Symptom: the install fails at render time with a duckdb: ... message naming a value.Cause: the chart validates its values before anything is created: the schedule must have five fields; tuning.memoryLimitPercent must be 20–80; minMemory/minCpu may not exceed their max counterparts; maxCpu:minCpu may not exceed 4:1; an objectStore.s3Compatible.endpoint may not carry a scheme; secretEnv[].name must be UPPER_SNAKE_CASE, unique and not one of the four AWS_* names; and the required fields for the chosen objectStore.type must be set.Fix: change the value the message names and install again. Nothing was created, so there is nothing to clean up.

The job never runs and cpln logs returns nothing

Symptom: the scheduled time passes, cpln logs shows no lines at all, and a manual cpln workload cron start produces no output either.Cause: a secret named in sql.secretName, secretEnv[] or objectStore.s3Compatible.credentialsSecretName does not exist, so the container cannot start.Fix: read status.versions[].message in cpln workload get-deployments RELEASE_NAME-duckdb --gvc GVC_NAME -o yaml, create the missing secret, then run cpln workload force-redeployment RELEASE_NAME-duckdb --gvc GVC_NAME.

Important Notes

This is a batch job, not a query service. Nothing is listening after install; that is correct behavior. For an always-on SQL endpoint, use trino.
Check both success signals — the duckdb-job-complete marker and a run status of Successful. Neither one alone is reliable.
activeDeadlineSeconds terminates a run within roughly two minutes of the deadline, not exactly at it, and the container bills for that window.
Every job must fit in memory. No volume is attached, so raise resources.maxMemory rather than relying on spill.
Never SET memory_limit higher than the container. Change tuning.memoryLimitPercent instead.
No .duckdb file is kept. Every run starts from an empty in-memory database; write results to object storage or an attached database.
Extensions are downloaded on every run from extensions.duckdb.org:443. Every execution depends on that host being reachable.
Prerequisite secrets must exist before install, and uninstalling the release does not delete them — the template only removes the secrets it created itself.
A cron workload runs in every location of its GVC. Install into a single-location GVC unless you want one run per location per schedule.
Installing this template several times is scale-out, not high availability. There is no failover and a failed run is not retried.
One script per install, by design. Multi-step, conditional or retrying pipelines belong in airflow.

External References

DuckDB Documentation

Official DuckDB documentation

Configuration Reference

Every setting available to a preamble or script

S3 API Support

Reading and writing s3:// paths with httpfs

Tuning Workloads

How memory and thread settings affect performance

DuckDB Template

View the source files, default values, and chart definition

Template Catalog

Templates

Overview

What Gets Created

Prerequisites

Installation

UI

CLI

Terraform

Pulumi

Configuration

Image and Schedule

SQL Script

Credentials in SQL

Resources and Tuning

Object Storage

Connecting

Operations

Confirming a Run Succeeded

Scaling and Availability

Troubleshooting

Important Notes

External References

DuckDB Documentation

Configuration Reference

S3 API Support

Tuning Workloads

DuckDB Template

​Overview

​What Gets Created

​Prerequisites

​Installation

UI

CLI

Terraform

Pulumi

​Configuration

​Image and Schedule

​SQL Script

​Credentials in SQL

​Resources and Tuning

​Object Storage

​Connecting

​Operations

​Confirming a Run Succeeded

​Scaling and Availability

​Troubleshooting

​Important Notes

​External References

DuckDB Documentation

Configuration Reference

S3 API Support

Tuning Workloads

DuckDB Template

Overview

What Gets Created

Prerequisites

Installation

Configuration

Image and Schedule

SQL Script

Credentials in SQL

Resources and Tuning

Object Storage

Connecting

Operations

Confirming a Run Succeeded

Scaling and Availability

Troubleshooting

Important Notes

External References