Spark

Overview

Spark deploys an Apache Spark standalone cluster — a distributed engine for batch ETL, large-scale data transformation, and SQL/DataFrame analytics. The template provisions a master (cluster manager and Web UI), a horizontally-scalable worker tier, an optional Spark Connect gRPC server for thin/remote clients, and an optional History Server that reads completed applications’ event logs back from an S3 or GCS bucket.

Architecture

Master — cluster manager and Web UI on :8080, cluster RPC on :7077. Fixed single replica.
Worker — the executor host tier, with its own Web UI on :8081. workers.replicas (default 1); more than one forms a multi-worker cluster.
Spark Connect (optional) — a gRPC server on :15002 for thin/remote client submission. Off by default.
History Server (optional) — a Web UI on :18080 that lists completed applications by reading their event logs back from object storage. Off by default; requires an S3/GCS bucket.

What Gets Created

Standard Spark Master Workload — {release}-spark-master, the cluster manager and Web UI. One replica; UI port :8080 (http), RPC port :7077 (tcp).
Standard Spark Worker Workload — {release}-spark-worker, the executor host tier. workers.replicas replicas; Web UI :8081 and RPC :7078 (tcp, internal only).
Standard Spark Connect Workload (optional) — {release}-spark-connect, a gRPC server on :15002 for remote client submission. Created only when connect.enabled: true.
Standard History Server Workload (optional) — {release}-spark-history, the completed-application dashboard on :18080. Created only when historyServer.enabled: true.
Config Secret — {release}-spark-conf, an opaque secret (encoding: plain) holding the rendered spark-defaults.conf (reverse-proxy, pinned RPC ports, and event-log/S3A settings). Always created.
Identity & Policy — one shared identity with a policy granting reveal on the config secret. When the History Server is enabled, the identity also carries the cloud-account binding for keyless object-storage access.

This template does not create a GVC. You must deploy it into an existing GVC. No persistent volume is created either — worker scratch is ephemeral, and the History Server’s durable store is the object-storage bucket.

Prerequisites

Default install: none. The master and a single worker come up with no external dependencies.
History Server (historyServer.enabled: true): an object-storage bucket and a Control Plane cloud account — AWS S3 or GCS. See Storage Setup.

Install the template using your preferred method:

UI

Browse, install, and manage templates visually

CLI

Manage templates from your terminal

Terraform

Declare templates in your Terraform configurations

Pulumi

Declare templates in your Pulumi programs

Configuration

The default values.yaml for this template:

# ─── Image ───────────────────────────────────────────────────────
image: apache/spark:4.0.4-scala2.13-java17-python3-ubuntu

# ─── Master (cluster manager + Web UI :8080) ─────────────────────
master:
  minCpu: 250m
  maxCpu: 1
  minMemory: 512Mi
  maxMemory: 1Gi

# ─── Workers (executor host tier) ────────────────────────────────
workers:
  replicas: 1        # 1 = proven single shape; >1 forms a multi-worker cluster (see README)
  cores: 2           # SPARK_WORKER_CORES — cores this worker offers to the cluster
  memory: 2g         # SPARK_WORKER_MEMORY — memory offered to executors (keep < maxMemory)
  minCpu: 500m
  maxCpu: 2
  minMemory: 1Gi
  maxMemory: 4Gi

# ─── Spark Connect (optional remote submission, gRPC :15002) ─────
connect:
  enabled: false
  minCpu: 250m
  maxCpu: 1
  minMemory: 512Mi
  maxMemory: 1Gi

# ─── History Server (persistent completed-app dashboard, :18080) ──
# Requires object storage (bucket + cloud account) — see README "Storage setup".
historyServer:
  enabled: false
  minCpu: 250m
  maxCpu: 1
  minMemory: 512Mi
  maxMemory: 1Gi

# ─── Object storage (required when historyServer.enabled) ────────
# Keyless (UCI): create a cpln cloud account + a bucket-scoped IAM policy; no keys in values.
storage:
  provider: aws              # aws | gcp
  bucket: my-spark-events    # bucket you created for event logs
  prefix: spark/events       # folder within the bucket
  region: us-east-1          # aws only
  cloudAccountName: my-spark-cloud-account   # cpln cloud account linked to your provider
  policyName: my-spark-events-policy         # aws only: bucket-scoped IAM policy name

# ─── Extra Spark configuration (merged into spark-defaults.conf) ──
sparkDefaults: {}
  # spark.sql.shuffle.partitions: "200"

# ─── Access ──────────────────────────────────────────────────────
internalAccess:
  type: same-gvc      # same-gvc (recommended) | same-org | workload-list
  # workloads:        # only when type is workload-list
  #   - //gvc/GVC/workload/WORKLOAD

publicAccess:
  enabled: false      # true exposes the Master + History Web UIs publicly — NO AUTH; see README

Image

image — The Spark container image used by every tier. The chart is shipped and tested on apache/spark:4.0.4-scala2.13-java17-python3-ubuntu.

Resources

Each tier (master, workers, connect, historyServer) exposes its own minCpu / maxCpu / minMemory / maxMemory. maxCpu maps to the workload’s CPU limit and maxMemory to its memory limit, with minCpu / minMemory as the reservation.

Workers

workers.replicas — Number of worker replicas. 1 is the proven single shape; a value greater than 1 forms a multi-worker cluster and tasks are spread across replicas.
workers.cores — Cores each worker offers to the cluster (SPARK_WORKER_CORES).
workers.memory — Memory each worker offers to executors (SPARK_WORKER_MEMORY). This is Spark’s offer to executors, not the container limit — keep it below workers.maxMemory.

Spark Connect

connect.enabled — When true, starts a Spark Connect gRPC server on :15002 for thin/remote client submission. Off by default.

A PySpark Connect client needs its own dependencies. The apache/spark image ships only the server side — install pyspark[connect] (or pandas, pyarrow, and grpcio) in the client environment.

History Server

historyServer.enabled — When true, starts the History Server on :18080, which lists completed applications by reading their event logs back from object storage. Off by default; it requires storage.* and a real bucket. See Storage Setup.

Object Storage

Configured only when the History Server is enabled. Access is keyless (UCI) — the workload identity federates with your cloud account and no access keys are stored in values.

Field	Description
`storage.provider`	`aws` or `gcp`.
`storage.bucket`	The bucket you created for event logs.
`storage.prefix`	Folder within the bucket where event logs are written.
`storage.region`	Bucket region. AWS only.
`storage.cloudAccountName`	The Control Plane cloud account linked to your provider.
`storage.policyName`	The bucket-scoped IAM policy name. AWS only.

The cluster’s driver tiers write event logs to this bucket and the History Server reads them back. See Storage Setup for the per-provider steps.

Extra Spark Configuration

sparkDefaults — A map of key: value entries merged verbatim into spark-defaults.conf, for any Spark setting the template does not expose directly (for example spark.sql.shuffle.partitions). See the Spark configuration reference.

Access

internalAccess.type — Controls which workloads may reach the cluster over the GVC network:

Type	Description
`same-gvc`	All workloads in the same GVC (recommended).
`same-org`	All workloads in the same organization.
`workload-list`	Only the workloads listed in `internalAccess.workloads`.

internalAccess.type: workload-list must include the cluster’s own workloads — the master, workers, and Connect server address each other over the GVC network, so an incomplete list breaks the cluster itself. same-gvc (the default) avoids this.

publicAccess.enabled — When true, exposes the Master and History Server Web UIs on a public *.cpln.app canonical endpoint. The worker and Connect internal UIs stay internal and are reached through the master’s reverse proxy. See Important Notes — the UIs are unauthenticated.

Submitting Jobs

Submit from the master container or any client workload in the GVC. A driver must advertise its pod IP — set spark.driver.host, or executors cannot connect back to the driver and the job stalls relaunching executors:

cpln workload exec RELEASE_NAME-spark-master --gvc GVC_NAME --container spark-master -- bash -c '
  export SPARK_LOCAL_IP=$(hostname -i | awk "{print \$1}")
  /opt/spark/bin/spark-submit \
    --master spark://RELEASE_NAME-spark-master.GVC_NAME.cpln.local:7077 \
    --conf spark.driver.host=$SPARK_LOCAL_IP \
    --class org.apache.spark.examples.SparkPi \
    /opt/spark/examples/jars/spark-examples_2.13-4.0.4.jar 20'

A successful SparkPi run prints Pi is roughly 3.14.... For remote/thin-client submission without an exec shell, enable Spark Connect and point a client at sc://…:15002.

Connecting

All endpoints are internal GVC addresses. Replace RELEASE_NAME and GVC_NAME with your values.

Target	Endpoint	Notes
Cluster RPC (submit)	`RELEASE_NAME-spark-master.GVC_NAME.cpln.local:7077`	`spark://…:7077`; internal.
Master Web UI	`RELEASE_NAME-spark-master.GVC_NAME.cpln.local:8080`	Worker and application UIs are proxied behind it.
Spark Connect	`sc://RELEASE_NAME-spark-connect.GVC_NAME.cpln.local:15002`	When `connect.enabled: true`.
History Server	`RELEASE_NAME-spark-history.GVC_NAME.cpln.local:18080`	When `historyServer.enabled: true`.

The worker and running-application (:4040) UIs are not exposed directly — the master serves them through its reverse proxy at /proxy/{id}/, so they are reachable through the master endpoint.

Reaching a Private UI

With publicAccess: false (the default), reach any Web UI in a browser with cpln port-forward:

cpln port-forward RELEASE_NAME-spark-master 8080:8080 --gvc GVC_NAME

Then open http://localhost:8080. The History Server is reached the same way — cpln port-forward RELEASE_NAME-spark-history 18080:18080 --gvc GVC_NAME. There are no credentials: Spark’s Web UIs and cluster port are unauthenticated.

Storage Setup

The History Server reads completed applications’ event logs from a bucket that the cluster’s driver tiers also write to. Access is keyless — the workload identity federates with your cloud account, so no keys are stored in values. Set historyServer.enabled: true and configure storage.*.

AWS S3
GCS

Create a bucket

Create your S3 bucket and set storage.bucket and storage.region to match. Set storage.provider: aws.

Set up a Cloud Account

If you do not have one, create a Cloud Account for your AWS account. Set storage.cloudAccountName to its name.

Create a bucket-scoped IAM policy

Create an AWS IAM policy with the JSON below (replace YOUR_BUCKET_NAME), then set storage.policyName to the policy’s name:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:ListBucket",
                "s3:GetBucketLocation",
                "s3:AbortMultipartUpload",
                "s3:ListBucketMultipartUploads",
                "s3:ListMultipartUploadParts"
            ],
            "Resource": [
                "arn:aws:s3:::YOUR_BUCKET_NAME",
                "arn:aws:s3:::YOUR_BUCKET_NAME/*"
            ]
        }
    ]
}

Set the prefix

Set storage.prefix to the folder within the bucket where event logs are written.

Create a bucket

Create your GCS bucket and set storage.bucket to its name. Set storage.provider: gcp.

Set up a Cloud Account

Create a Cloud Account for your GCP account. Set storage.cloudAccountName to its name.

Grant the Storage Admin role

Grant the GCP service account created for the Cloud Account the Storage Admin (roles/storage.objectAdmin) role on the bucket. Set storage.prefix to the folder for event logs. region and policyName are not used for GCP.

The History Server crash-loops if the bucket does not exist. Enable it only with a real bucket and cloud account — an unreachable or missing bucket surfaces UnknownStoreException / NoSuchBucket in the History Server log. An empty History Server list is normal until a job completes. The S3A connector jars are downloaded from Maven Central at container start on every driver tier and the History Server.

Important Notes

No authentication anywhere by default — every Web UI and the cluster port are open to whoever can reach them. Keep publicAccess: false and reach UIs via cpln port-forward. Setting publicAccess: true puts the unauthenticated Master and History UIs on the public internet.
A driver must set spark.driver.host to its pod IP (see Submitting Jobs) — without it the driver advertises a non-routable pod hostname and the job stalls relaunching executors.
The History Server needs an existing bucket — enable historyServer.enabled and point storage.* at a real bucket and cloud account, or it crash-loops. Access is keyless via the workload identity.
Spark Connect clients need their own dependencies — the apache/spark image ships only the server side. Install pyspark[connect] (or pandas, pyarrow, and grpcio) in the client environment.
Switching provider (aws ↔ gcp) needs a fresh install, not an upgrade — cloud-binding blocks on an identity are never cleared on update.
internalAccess.type: workload-list must include the cluster’s own workloads — the master, workers, and Connect server address each other over the GVC network. same-gvc (the default) avoids this.
Worker scratch is ephemeral and workers.memory is Spark’s offer to executors, not the container limit — keep it below workers.maxMemory. Master loss is minutes of downtime, not data loss (there is no Master HA in this version).

External References

Spark Standalone Mode

How the standalone master and workers operate

Monitoring & History Server

Event logs, the History Server, and metrics

Spark Connect

Thin/remote client submission over gRPC

Submitting Applications

spark-submit options and application packaging

Hadoop-AWS S3A

The S3A connector used for event-log storage

Spark Template

View the source files, default values, and chart definition

Template Catalog

Templates

Overview

Architecture

What Gets Created

Prerequisites

UI

CLI

Terraform

Pulumi

Configuration

Image

Resources

Workers

Spark Connect

History Server

Object Storage

Extra Spark Configuration

Access

Submitting Jobs

Connecting

Reaching a Private UI

Storage Setup

Important Notes

External References

Spark Standalone Mode

Monitoring & History Server

Spark Connect

Submitting Applications

Hadoop-AWS S3A

Spark Template

​Overview

​Architecture

​What Gets Created

​Prerequisites

UI

CLI

Terraform

Pulumi

​Configuration

​Image

​Resources

​Workers

​Spark Connect

​History Server

​Object Storage

​Extra Spark Configuration

​Access

​Submitting Jobs

​Connecting

​Reaching a Private UI

​Storage Setup

​Important Notes

​External References

Spark Standalone Mode

Monitoring & History Server

Spark Connect

Submitting Applications

Hadoop-AWS S3A

Spark Template

Overview

Architecture

What Gets Created

Prerequisites

Configuration

Image

Resources

Workers

Spark Connect

History Server

Object Storage

Extra Spark Configuration

Access

Submitting Jobs

Connecting

Reaching a Private UI

Storage Setup

Important Notes

External References