Overview
NATS is an open-source, high-performance, lightweight messaging system for pub/sub, queueing and request/reply, with an optional built-in persistence layer (JetStream) for durable streams, consumers, a key-value store and an object store. This template deploys a NATS cluster per location you list —replicas servers in each — and joins the locations into a cross-region super cluster over NATS gateways when you list more than one. A WebSocket listener is on by default; Control Plane serves it on port 443 once you open the public firewall.
locations must already be one of the GVC’s locations. A GVC location you do not list runs nothing — its deployment reads This workload location is deactivated because maxScale is set to 0, which is the expected result of pinning the location list, not a fault.What Gets Created
Prerequisites
A GVC that contains every location in locations
spec.staticPlacement.locationLinks is the list. Every name in locations must appear there; the GVC may have more locations than you list, and no NATS server runs in those. A declared location the GVC lacks is not caught at install time — each server reports it at boot (see Troubleshooting). To add a location to a GVC, see GVC Locations; every workload in that GVC then runs there, so add locations to a shared GVC deliberately.Installation
The defaultlocations list is a single location, aws-us-east-1, with three servers. If your GVC has that location, the defaults install as-is; otherwise put your GVC’s own location(s) in a values file first — locations is a list, so set it with -f rather than --set:
UI
CLI
Terraform
Pulumi
Configuration
Each subsection below shows the shipped default for one area ofvalues.yaml. Override only what you need with a values file or --set.
Locations and Cluster Sizing
locations[].name— A location of the GVC you install into. Listing the same location twice is rejected at render time.locations[].replicas— How many NATS servers run in that location; at least1. A location withreplicas: 0is rejected at render time — to stop running somewhere, remove the entry.
Image and Resources
image— The NATS container image. Bump the tag to upgrade NATS.resources.cpu/resources.memory— CPU and memory limit for each server. The defaults are small; raise them for real throughput.
Listeners
nats_defaults.port— Client port, fornats://clients inside the GVC.nats_defaults.cluster.port— Route connections between servers in the same location.nats_defaults.gateway.port— Gateway connections between locations. Published as a container port only when more than one location is configured.nats_defaults.websocket.enabled— Serves a WebSocket listener and publishes the workload’s public endpoint. Control Plane serves it on port 443; who can reach it is decided by Public Access.nats_defaults.websocket.port/compression/noTls— The internal WebSocket port, per-message compression, and whether the listener itself runs without TLS. LeavenoTls: trueunless you supply certificates throughnats_extra_config.
helm template: a reserved port is rejected by the API at install, and a port collision is accepted and then stops the container at boot with address already in use.
Public Access
allowCIDR— The inbound CIDR ranges allowed to reach the WebSocket endpoint from the internet. It applies to the WebSocket port only — the client (4222), route (6222) and gateway (7222) ports are never exposed publicly.
/32. A client behind a proxy pool or a rotating egress address moves to a different IP, and a perfectly healthy deployment then returns 403 to it. Use a range wide enough to cover your egress, not just the address you happen to have today. A firewall change takes a few minutes to take effect; re-test rather than trusting the first response.cpln port-forward — see Connecting.
JetStream
jetstream.enabled— Without JetStream, NATS is in-flight only: a message with no connected subscriber is gone. Enabling it adds durable streams, durable consumers, a key-value store and an object store, and creates the volume set. Sizing is decided by the total server count — see the table under Locations and Cluster Sizing; exactly two servers is refused.volumeset.capacity— Size in GiB of the persistent volume each server gets for/data/nats(minimum 10). Used only whenjetstream.enabledistrue. Uninstalling the release deletes the volume set and the JetStream data on it.
nats stream add STREAM --subjects "SUBJECT.>" --replicas=3 --defaults — which is per-stream, in your application, not a setting in this template.Extra NATS Configuration
nats_extra_config— Appended verbatim to the generated server configuration. Use it for anything the template does not expose — payload limits, accounts, authorization:
Internal Access
internalAccess.type— Controls which workloads can reach the servers from inside Control Plane.
internalAccess.workloads— The workload links allowed in whentypeisworkload-list. List only your clients.
Connecting
status.canonicalEndpoint; connect with wss:// on the default port.
Verify a server from inside its container. The image carries no nats CLI, but it does carry wget, and the monitoring listener answers on loopback:
/healthz returns {"status":"ok"} on a healthy server, and /routez lists the route connections to the other servers in the same location — an empty routes list on a multi-server deployment means the cluster has not formed (see Troubleshooting). On a multi-location release, /gatewayz lists the other locations as gateways.
To publish and subscribe from your own machine, forward the client port and point a local nats CLI at 127.0.0.1:
Operations
Backing Up
Without JetStream there is nothing to back up — messages are in flight only. Withjetstream.enabled: true, the data lives on the volume set RELEASE_NAME-nats-vs, one volume per server, and two independent layers can protect it:
-
Volume set snapshots. The template configures no snapshot schedule, so take them on demand, naming the location of the volume you want:
-
A stream snapshot. The
natsCLI can write a complete copy of one stream — every message, its configuration and, with--consumers, its consumers’ positions — to a local directory. Run it from your machine through a port-forward tunnel (see Connecting):Memory-storage streams cannot be snapshotted on this server version; only file-backed streams can. Follow the upstream stream backup and restore guide for the options.
nats backup validate ./backups/STREAM passes before you rely on it.Restoring a Backup
Restoring is not a procedure this template performs or has verified. Two upstream paths exist, and neither has been exercised against a live install of this template:- A stream snapshot is restored with
nats --server nats://127.0.0.1:4222 backup restore stream ./backups/STREAMagainst a JetStream-enabled cluster in which the stream does not already exist — restore recreates the stream and fails if one of that name is present. Rehearse it on a throwaway release first. - A volume set snapshot can in principle be restored with
cpln volumeset snapshot restore, but this template gives you no supported way to stop a server and replace/data/natsunderneath it, so treat that path as unverified and prefer the stream-level restore.
Migrating from 2.x
Template versions through 2.0.0 created their own GVC; 3.0.0 deploys into one you already have. Install 3.0.0 as a new release against an existing GVC, move the traffic, then remove the old release.Choose the target GVC
cpln gvc get GVC_NAME -o yaml — spec.staticPlacement.locationLinks is the list. Do not point the new release at the GVC the old release created: that GVC is still owned by the old Helm release, and uninstalling it later would take the new workloads with it.Install 3.0.0 as a new release
gvc.locations entries into the top-level locations list and install as shown in Installation. Note that allowCIDR now defaults to closed — if the 2.x deployment served browser clients over WebSocket, set your own ranges explicitly, together with an authorization { ... } block.Move the traffic
nats CLI as described in Backing Up and Restoring a Backup.Remove the old release
Scaling and Availability
- Fault tolerance follows the total server count. Core NATS keeps routing with any number of servers up. JetStream keeps serving while a majority of servers is up: 3 survive losing one, 5 survive losing two, and 2 survive none, which is why that shape is refused (see Locations and Cluster Sizing).
- A stream survives a server only if you created it with replicas. Stream data is placed per stream at creation time — a default
R1stream lives on one server and is lost with it, whatever the cluster size. Create streams with--replicas=3on a 3-or-more-server cluster. - Losing a whole location takes that location’s cluster with it. Clients connected to the other locations keep working, and streams with enough replicas in the surviving locations keep serving — give each location at least 3 servers if a location’s loss must not take a stream down.
- Change
locations[].replicaswith acpln helm upgrade. The servers restart to pick up the new route list. With JetStream, do not pass through the 2-server shape — the chart refuses it at render time, and a 3-server roster whose GVC is missing a location becomes it at runtime. - A server rescheduled with its volume intact rejoins on its own. Each server derives its name, cluster and peers from the location and replica it runs as, so the configuration is stable across restarts.
Troubleshooting
When a server will not come up or a cluster will not form, its own[nats] log lines name the reason:
Install or upgrade fails: the gvc values key was REMOVED
Install or upgrade fails: the gvc values key was REMOVED
gvc block. Version 3.0.0 no longer creates a GVC and refuses to render rather than let an in-place upgrade delete the one the old release created.Fix: Do not upgrade in place. Follow Migrating from 2.x: install 3.0.0 as a new release with a top-level locations list, move the traffic, then uninstall the old release.Install fails: locations must be a LIST of name and replicas entries
Install fails: locations must be a LIST of name and replicas entries
--set locations=[] (or any --set of the whole key) sets a string, not a list. The chart rejects it before any resource is created.Fix: Put locations in a values file and pass it with -f, as shown in Installation. Indexed overrides such as --set 'locations[0].replicas=5' on top of a values file are fine.Install fails: JetStream on exactly 2 servers has no fault tolerance and is worse than 1
Install fails: JetStream on exactly 2 servers has no fault tolerance and is worse than 1
jetstream.enabled is true and replicas across all locations adds up to 2. The JetStream meta group’s quorum would be 2 of 2, so losing either server takes JetStream down completely.Fix: Use 1 server (standalone JetStream) or at least 3 in total.Install fails: internalAccess.type none closes the internal path the replicas use to reach EACH OTHER
Install fails: internalAccess.type none closes the internal path the replicas use to reach EACH OTHER
internalAccess.type: none with more than one server would leave every server running alone — routes never establish and JetStream gets no meta group.Fix: Use same-gvc, or workload-list naming your clients (the template adds its own workload to the list). none is only valid for a single server reached over the public WebSocket endpoint.Install fails: port is reserved for system use, or port is used by both
Install fails: port is reserved for system use, or port is used by both
nats_defaults ports is a port Control Plane reserves (8012, 8022, 9090, 9091, 15000, 15001, 15006, 15020, 15021, 15090, 41000), or two listeners — including the fixed monitoring port 8222 — share a port.Fix: Pick another port for the listener the error names. Both cases are caught at render time because they are invisible to helm template otherwise.Server exits with [nats] FATAL: refusing to bootstrap JetStream on a cluster smaller than it was configured to be
Server exits with [nats] FATAL: refusing to bootstrap JetStream on a cluster smaller than it was configured to be
1; its log reads locations declared in values are not in GVC '...', lists what the GVC has, and says how many of the declared servers can actually start.Cause: locations names a location the GVC does not have. The platform accepts and stores it without any error, so the servers declared there never start, and every running server keeps retrying route and gateway URLs that can never answer. A server with an empty JetStream store refuses to bootstrap a cluster smaller than configured; a server whose store already holds data logs the same lines as a warning and keeps serving, so the check can never take a live cluster down. With JetStream off, it only ever warns.Fix: Add the missing location to the GVC, or remove it from locations, and redeploy. cpln gvc get GVC_NAME -o yaml shows the GVC’s list under spec.staticPlacement.locationLinks.Server exits with [nats] FATAL: this server is running in location ..., which is not in the configured location list
Server exits with [nats] FATAL: this server is running in location ..., which is not in the configured location list
locations does not list. Such a server would name its cluster after its own location, so no other server has a route to it and no gateway names it — it would accept clients into a cluster of one and silently drop what they publish. It exits instead, fresh or not.Fix: Add that location to locations and upgrade, or remove it from the GVC. Normally an unlisted GVC location runs nothing at all; this line only appears if a replica was placed there anyway.Log shows [nats] WARNING: could not read the location list of GVC
Log shows [nats] WARNING: could not read the location list of GVC
RELEASE_NAME-nats-gvc-policy policy is missing (the log then adds A 403 here means the GVC-read policy is missing) — so it skipped the location check and started anyway. NATS runs normally.Fix: Nothing, if the warning was transient. If it recurs on every start, confirm the policy exists and grants view on the GVC the release is installed into:Log shows [nats] WARNING: exactly 2 servers are running
Log shows [nats] WARNING: exactly 2 servers are running
locations adds up to more than 2, but only 2 declared servers can actually start — typically because one location is missing from the GVC. At runtime this is the 2-server shape the chart refuses in values: with JetStream enabled its quorum is 2 of 2, and without JetStream the warning tells you not to enable it on this shape.Fix: Resolve the missing location as in the FATAL entry above so the full roster runs, or change locations to a 1- or 3-plus-server total.Container restarts immediately after setting nats_extra_config
Container restarts immediately after setting nats_extra_config
nats-server, which refuses to start and prints its own parse error.Fix: Read the server’s error and correct the block:Servers are ready but /routez shows no routes
Servers are ready but /routez shows no routes
ready: true says nothing about clustering — this workload has no readiness probe. An empty routes list on a multi-server deployment means the servers cannot reach each other on the route port: internalAccess is workload-list with a list that was edited outside the chart, or a declared location is missing from the GVC so the peers never started.Fix: Check the [nats] log lines with the command at the top of this section, confirm internalAccess as shown in Internal Access, and give a firewall change a few minutes to take effect before re-testing.WebSocket clients get 403 from the public endpoint
WebSocket clients get 403 from the public endpoint
allowCIDR is empty (the default), or does not cover the client’s current egress address — a client behind a proxy pool or a rotating address moves to an IP outside a narrow range.Fix: Set allowCIDR to a range that covers your clients’ egress, together with an authorization { ... } block in nats_extra_config, and upgrade the release. Allow a few minutes for the change to take effect.Important Notes
- Do not
cpln helm upgradea 2.x release onto this version. It deletes the GVC the old release created and everything inside it. Follow Migrating from 2.x. locationsmust match the GVC’s locations. A declared location the GVC lacks is not caught at install; a fresh JetStream server refuses to start on it, and every other server warns. Read the GVC’s list withcpln gvc get GVC_NAME -o yamlbefore installing.- A GVC location you did not list runs nothing. Its deployment reads
This workload location is deactivated because maxScale is set to 0; that is expected. - No public access by default, and no authentication. Opening
allowCIDRwithout anauthorization { ... }block innats_extra_configpublishes an unauthenticated message bus. Never pin a single/32. internalAccessgoverns the servers’ traffic to each other. Withworkload-list, list only your clients — the template adds its own workload.noneis refused for more than one server.- JetStream data lives on the volume set and is deleted when you uninstall. Streams default to one replica; create them with
--replicas=3if a stream must survive a server loss. - Do not run JetStream on exactly two servers. The chart refuses it in values and warns if a missing GVC location produces it at runtime.
nats_extra_configis injected verbatim, so a syntax error there stops the container rather than failing the render, and anything you put in it is stored in the release values.- A firewall change takes a few minutes to take effect. After changing
allowCIDRorinternalAccess, keep re-testing rather than concluding the knob is broken.