Spark

Overview

Apache Spark is a distributed engine for batch ETL, large-scale data transformation, and SQL and DataFrame analytics. This template runs a Spark standalone cluster: a master (cluster manager and Web UI), a worker tier you scale with one value, an optional Spark Connect gRPC server for remote clients, and an optional History Server that lists completed applications by reading their event logs back from an S3 or GCS bucket.

This template deploys into an existing GVC that you already have; it does not create or manage one. Every workload runs in each location of that GVC, and the template was validated in a single-location GVC, so install it into one.

What Gets Created

Resource	Name	Purpose	Optional
Workload	`RELEASE_NAME-spark-master`	Cluster manager and Web UI (standard, 1 replica; Web UI `8080`, cluster RPC `7077`)	—
Workload	`RELEASE_NAME-spark-worker`	Executor host tier (standard, `workers.replicas` replicas; Web UI `8081`)	—
Workload	`RELEASE_NAME-spark-connect`	Spark Connect gRPC server on `15002`	`connect.enabled: true`
Workload	`RELEASE_NAME-spark-history`	History Server Web UI on `18080`, reading event logs from your bucket	`historyServer.enabled: true`
Secret	`RELEASE_NAME-spark-conf`	Rendered `spark-defaults.conf` mounted into every workload (reverse proxy, pinned RPC ports, event-log settings, your `sparkDefaults`)	—
Identity	`RELEASE_NAME-spark-identity`	Shared by all workloads; carries the AWS or GCP cloud account binding when the History Server is on	—
Policy	`RELEASE_NAME-spark-policy`	Grants the identity `reveal` on the config secret only	—

No volume set is created. Worker scratch space is ephemeral, and the only durable store is the bucket the History Server reads.

Prerequisites

None for a default install. The master and a single worker start with no external dependencies. The History Server (historyServer.enabled: true) needs an object-storage bucket and a Control Plane cloud account before you enable it. Access is keyless: the workload identity federates with the cloud account, so no access keys are created or stored.

Bucket and cloud account for the History Server

AWS S3
GCS

Create an S3 bucket for event logs. Set storage.provider: aws, storage.bucket to its name and storage.region to its region.
If you do not have one, create a cloud account for your AWS account. Set storage.cloudAccountName to its name.
Create an AWS IAM policy scoped to the bucket (replace YOUR_BUCKET_NAME), then set storage.policyName to the policy name:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:ListBucket",
                "s3:GetBucketLocation",
                "s3:AbortMultipartUpload",
                "s3:ListBucketMultipartUploads",
                "s3:ListMultipartUploadParts"
            ],
            "Resource": [
                "arn:aws:s3:::YOUR_BUCKET_NAME",
                "arn:aws:s3:::YOUR_BUCKET_NAME/*"
            ]
        }
    ]
}

Optionally change storage.prefix, the folder inside the bucket where event logs are written.

Create a GCS bucket for event logs. Set storage.provider: gcp and storage.bucket to its name.
Create a cloud account for the GCP project that holds the bucket. Set storage.cloudAccountName to its name.
Grant the cloud account’s service account the Storage Admin (roles/storage.admin) role. The template additionally binds the workload identity to roles/storage.objectAdmin on exactly the bucket in storage.bucket.
Optionally change storage.prefix. storage.region and storage.policyName are not used for GCP.

The bucket must exist before you enable the History Server. If it is missing or unreachable, the History Server crash-loops with UnknownStoreException or NoSuchBucket in its log.

Installation

A default install needs no --set values:

cpln helm install RELEASE_NAME oci://ghcr.io/controlplane-com/templates/spark \
  --version 1.0.0 --gvc GVC_NAME

Or follow the instructions for your preferred method:

UI

Browse, install, and manage templates visually

CLI

Manage templates from your terminal

Terraform

Declare templates in your Terraform configurations

Pulumi

Declare templates in your Pulumi programs

Configuration

Image

image: apache/spark:4.0.4-scala2.13-java17-python3-ubuntu

One image runs every tier.

Master

master:
  minCpu: 250m
  maxCpu: 1
  minMemory: 512Mi
  maxMemory: 1Gi

The master is always a single replica. maxCpu and maxMemory are the container limits; minCpu and minMemory are the reservation. The same four keys appear on every tier.

Workers

workers:
  replicas: 1        # 1 = proven single shape; >1 forms a multi-worker cluster (see README)
  cores: 2           # SPARK_WORKER_CORES — cores this worker offers to the cluster
  memory: 2g         # SPARK_WORKER_MEMORY — memory offered to executors (keep < maxMemory)
  minCpu: 500m
  maxCpu: 2
  minMemory: 1Gi
  maxMemory: 4Gi

workers.memory is what each worker offers to executors, not the container limit. Keep it below workers.maxMemory, which also has to cover the worker process itself.

Spark Connect

connect:
  enabled: false
  minCpu: 250m
  maxCpu: 1
  minMemory: 512Mi
  maxMemory: 1Gi

connect.enabled: true starts a Spark Connect server on gRPC port 15002, so thin clients can run DataFrame and SQL work without an exec shell. See Using Spark Connect.

History Server

historyServer:
  enabled: false
  minCpu: 250m
  maxCpu: 1
  minMemory: 512Mi
  maxMemory: 1Gi

historyServer.enabled: true starts the History Server on 18080 and turns on event logging for applications that use the cluster’s spark-defaults.conf. It requires the bucket and cloud account from Prerequisites and the storage values below. While it is enabled, every workload downloads the S3 or GCS connector jars from Maven Central at start, which needs outbound internet access.

Object Storage

storage:
  provider: aws              # aws | gcp
  bucket: my-spark-events    # bucket you created for event logs
  prefix: spark/events       # folder within the bucket
  region: us-east-1          # aws only
  cloudAccountName: my-spark-cloud-account   # cpln cloud account linked to your provider
  policyName: my-spark-events-policy         # aws only: bucket-scoped IAM policy name

Used only when historyServer.enabled is true. The chart refuses to render when the History Server is on and storage.bucket or storage.cloudAccountName is empty, or storage.provider is not aws or gcp.

Extra Spark Configuration

sparkDefaults: {}
  # spark.sql.shuffle.partitions: "200"

Each key: value pair is appended to the rendered spark-defaults.conf, for any Spark setting the template does not expose directly.

Access

internalAccess:
  type: same-gvc      # same-gvc (recommended) | same-org | workload-list
  # workloads:        # only when type is workload-list
  #   - //gvc/GVC/workload/WORKLOAD

publicAccess:
  enabled: false      # true exposes the Master + History Web UIs publicly — NO AUTH; see README

`internalAccess.type`	Who can reach the cluster over the GVC network
`same-gvc`	Every workload in the same GVC (recommended)
`same-org`	Every workload in the organization
`workload-list`	Only the workloads in `internalAccess.workloads`

With workload-list, the list must include the cluster’s own workloads. The master, workers and Connect server talk to each other over the GVC network, so leaving any of them out breaks the cluster. same-gvc avoids this.

publicAccess.enabled: true puts the Master and History Server Web UIs on public *.cpln.app endpoints. Spark’s Web UIs have no authentication, so anyone with the URL can see the cluster and its applications. Workers and the Connect server are never exposed publicly. A firewall change can take a few minutes to take effect.

Connecting

Spark’s Web UIs and cluster port are unauthenticated, so there are no credentials to retrieve.

What	Address	Notes
Cluster RPC	`spark://RELEASE_NAME-spark-master.GVC_NAME.cpln.local:7077`	The `--master` URL for `spark-submit`; internal only
Master Web UI	`RELEASE_NAME-spark-master.GVC_NAME.cpln.local:8080`	Worker and running-application UIs are served behind it at `/proxy/`
Spark Connect	`sc://RELEASE_NAME-spark-connect.GVC_NAME.cpln.local:15002`	When `connect.enabled: true`; internal only
History Server	`RELEASE_NAME-spark-history.GVC_NAME.cpln.local:18080`	When `historyServer.enabled: true`
Public Web UIs	`status.canonicalEndpoint` of the master or history workload	When `publicAccess.enabled: true`

Read a public endpoint with:

cpln workload get RELEASE_NAME-spark-master --gvc GVC_NAME -o yaml

To verify an install, forward the master Web UI to your machine and open http://localhost:8080 — the worker should be listed as ALIVE:

cpln port-forward RELEASE_NAME-spark-master 8080:8080 --gvc GVC_NAME

Submitting Jobs

Submit from the master container or from any client workload in the GVC. The driver must advertise its own IP with spark.driver.host; without it, executors cannot connect back and the application keeps relaunching executors.

cpln workload exec RELEASE_NAME-spark-master --gvc GVC_NAME --container spark-master -- bash -c '
  export SPARK_LOCAL_IP=$(hostname -i | awk "{print \$1}")
  /opt/spark/bin/spark-submit \
    --master spark://RELEASE_NAME-spark-master.GVC_NAME.cpln.local:7077 \
    --conf spark.driver.host=$SPARK_LOCAL_IP \
    --class org.apache.spark.examples.SparkPi \
    /opt/spark/examples/jars/spark-examples_2.13-4.0.4.jar 20'

A successful run prints Pi is roughly 3.14. A job submitted from the template’s own containers picks up the mounted spark-defaults.conf, so with the History Server enabled its event log lands in your bucket. A driver running elsewhere has to set its own event-log settings to appear in the History Server. With publicAccess.enabled: true, add --conf spark.ui.reverseProxyUrl= followed by the master’s public URL to your submit command, so the links in the application’s own UI resolve in a browser.

Using Spark Connect

With connect.enabled: true, a PySpark client in any workload of the GVC connects with the sc:// address. The apache/spark image ships only the server side, so install the client package — it pulls in pandas, pyarrow and grpcio — in the client environment:

pip install 'pyspark[connect]'

from pyspark.sql import SparkSession

spark = SparkSession.builder.remote("sc://RELEASE_NAME-spark-connect.GVC_NAME.cpln.local:15002").getOrCreate()
print(spark.range(1000).count())

Match the client’s PySpark version to the server’s Spark version (4.0).

Reaching a Private UI

With publicAccess.enabled: false (the default), open any Web UI through cpln port-forward, which works with the firewall fully closed. The History Server is reached the same way as the master:

cpln port-forward RELEASE_NAME-spark-history 18080:18080 --gvc GVC_NAME

Then open http://localhost:18080. An empty application list is normal until the first job with event logging completes.

Operations

Scaling and Availability

Change the worker count with workers.replicas and a Helm upgrade; each worker registers with the master and offers workers.cores and workers.memory to executors:

cpln helm upgrade RELEASE_NAME oci://ghcr.io/controlplane-com/templates/spark \
  --version 1.0.0 --gvc GVC_NAME \
  --set workers.replicas=3

The master is a single replica with no high availability in this version. It keeps no data, so losing it costs downtime, not data. Completed-application history survives in the bucket. Switching storage.provider between aws and gcp needs a fresh install, not an upgrade — the identity keeps its old cloud binding when it is updated. Uninstall, then install again with the new provider:

cpln helm uninstall RELEASE_NAME --gvc GVC_NAME

Troubleshooting

A job never finishes and keeps relaunching executors

Symptom: the application appears in the master UI, but executors are repeatedly launched and lost and the job makes no progress.Cause: the driver advertises an address the executors cannot reach, so they never connect back to it.Fix: set spark.driver.host to the driver’s own IP on submit, as in Submitting Jobs. This looks like a resource problem but is a networking one.

The History Server crash-loops with NoSuchBucket

Symptom: RELEASE_NAME-spark-history never becomes ready, and its log shows UnknownStoreException or NoSuchBucket.Cause: the bucket in storage.bucket does not exist, or the identity cannot reach it.Fix: create the bucket and check the cloud account and IAM policy or role from Prerequisites. Read the log with:

cpln logs '{gvc="GVC_NAME", workload="RELEASE_NAME-spark-history"}' --limit 50 --since 10m

A Spark Connect client fails to import pandas or pyarrow

Symptom: SparkSession.builder.remote(...) raises an import error for pandas, pyarrow or grpcio.Cause: a Spark Connect client needs its own client-side dependencies; the server image does not provide them to remote clients.Fix: install pyspark[connect] in the client environment.

Workers stop registering after switching to workload-list

Symptom: after setting internalAccess.type: workload-list, the cluster stops working — workers are missing from the master UI or jobs never get executors.Cause: the cluster’s own workloads are not in internalAccess.workloads, so the firewall blocks them from each other.Fix: add the master, worker and (if enabled) Connect workloads to the list, or go back to same-gvc.

Links in a public application UI point to a cpln.local address

Symptom: with publicAccess.enabled: true, a link inside a running application’s UI opens a *.cpln.local address that does not resolve in your browser.Cause: the application was submitted without the master’s public URL, so it builds its links from the internal hostname.Fix: add --conf spark.ui.reverseProxyUrl= with the master’s public URL when you submit the job.

Important Notes

Nothing is authenticated. Keep publicAccess.enabled: false and use cpln port-forward; turning it on puts the Master and History UIs on the public internet with no login.
Every driver must set spark.driver.host to its own IP, or its executors cannot connect back.
Create the bucket before enabling the History Server, or it crash-loops.
Switching storage.provider needs a fresh install, not a Helm upgrade.
With internalAccess.type: workload-list, include the cluster’s own workloads in the list.
Keep workers.memory below workers.maxMemory — it is Spark’s offer to executors, not the container limit.
Install into a single-location GVC — every workload runs in each location the GVC has.

External References

Spark Standalone Mode

How the standalone master and workers operate

Monitoring and History Server

Event logs, the History Server, and metrics

Spark Connect

Thin and remote client access over gRPC

Submitting Applications

spark-submit options and application packaging

Spark Template

View the source files, default values, and chart definition

Template Catalog

Templates

Overview

What Gets Created

Prerequisites

Installation

UI

CLI

Terraform

Pulumi

Configuration

Image

Master

Workers

Spark Connect

History Server

Object Storage

Extra Spark Configuration

Access

Connecting

Submitting Jobs

Using Spark Connect

Reaching a Private UI

Operations

Scaling and Availability

Troubleshooting

Important Notes

External References

Spark Standalone Mode

Monitoring and History Server

Spark Connect

Submitting Applications

Spark Template

​Overview

​What Gets Created

​Prerequisites

​Installation

UI

CLI

Terraform

Pulumi

​Configuration

​Image

​Master

​Workers

​Spark Connect

​History Server

​Object Storage

​Extra Spark Configuration

​Access

​Connecting

​Submitting Jobs

​Using Spark Connect

​Reaching a Private UI

​Operations

​Scaling and Availability

​Troubleshooting

​Important Notes

​External References

Spark Standalone Mode

Monitoring and History Server

Spark Connect

Submitting Applications

Spark Template

Overview

What Gets Created

Prerequisites

Installation

Configuration

Image

Master

Workers

Spark Connect

History Server

Object Storage

Extra Spark Configuration

Access

Connecting

Submitting Jobs

Using Spark Connect

Reaching a Private UI

Operations

Scaling and Availability

Troubleshooting

Important Notes

External References